OCaml.org blog

tarides — 2023-05-02T14:42:52-00:00

MinidebConf TN 23 was organised by Debian Developers and Villupuram Linux Users Group (VGLUG) as a precursor to DebConf 23 in September at Kochi, India. I had an opportunity to attend and speak at MiniDebConf TN.

+https://ocaml.org/feed.xmlOCaml.org blog2023-05-02T14:42:52-00:00https://tarides.com/feed.xmltarides

I presented two sessions, one built on our experiences of introducing a Code of Conduct to an open source community (slides here), and one called An Invitation to OCaml, aimed at people with no prior OCaml experience. I was pleased to see a lot of folks getting interested in learning OCaml.

Over the course of two days, I attended interesting sessions by speakers from across India and other parts of the world.

First Day

diff --git a/data/planet/ahrefs.xml b/data/planet/ahrefs.xml new file mode 100644 index 0000000000..a7559d7b8b --- /dev/null +++ b/data/planet/ahrefs.xml @@ -0,0 +1,2 @@ + +https://medium.com/feed/ahrefs/tagged/ocamlahrefs2023-05-02T14:42:52-00:00https://medium.com/feed/ahrefs/tagged/ocamlahrefs

Intro

There has never been a better time to learn OCaml, one of the premier statically-typed functional programming languages used in industry. We at Ahrefs have used it on our backend since the early days of the company, and since 2018 have even used it extensively for our frontend code. Today we’re very excited to share with you our favorite recommendations for getting started with OCaml!

Best resources for learning OCaml

If you have little functional programming experience or are even a beginner to programming, the place to start is OCaml from the Very Beginning, a book that is now free thanks to generous funding from the OCaml Software Foundation (which Ahrefs is a sponsor of). The book can be viewed directly from its website and can also be downloaded as a PDF. While advertised as being approachable even to new programmers, this doesn’t quite seem to be true, at least based on feedback we’ve received. The text itself introduces concepts in a structured way, but the exercises require a little background in programming to complete. Such background info can be obtained by watching the opening videos in this series of lectures from Cornell’s OCaml programming course. Speaking of the Cornell course, their official online textbook OCaml Programming: Correct + Efficient + Beautiful is an excellent resource for learners with some programming experience under their belt (specifically, you should have written some code using a mainstream imperative language like Python or Java).

How to install OCaml

The official installation instructions are entirely adequate. It shows you how to install the compiler and some useful dev tools like dune (build system), utop (interactive read-eval-print loop), and ocaml-lsp-server (useful for editor integration).

Actually, if you are working through OCaml from the Very Beginning, you do not need to install OCaml during the first several chapters, as you can execute code snippets directly on the Try OCaml page, or create a notebook at Sketch.sh, an interactive OCaml notebook site maintained by Ahrefs through our monthly Open Source Friday program.

Editor support

For most beginners to the language, we recommend the official OCaml Platform Visual Studio Code extension. The official OCaml installation guide has a good section on setting it up. There is also great support for users of emacs and vim.

Tips

Even early on, it’s a good idea to start saving your code into .ml files and learning how to run it. The easiest way to run a simple program is to start up utop and inside of it run #use "name_of_your_program.ml" as described here.

When creating a new notebook in Sketch.sh, the default syntax is ReasonML (it’s not a different language, just an alternate syntax that more resembles JavaScript). Click on ML in the top left corner to switch to the original OCaml syntax.

Conclusion

OCaml originated from French academia more than 25 years ago, and from there spread to elite universities and forward-thinking companies around the world. Now the OCaml community has produced high quality learning material that is both free and easy to access. So take the initiative and learn you some OCaml!

How to get started with OCaml in 2022 was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Monorobot: a Slack bot for monorepos

ahrefs — 2021-12-09T15:19:04-00:00

Monorobot: a notification bot for monorepos

Monorobot enables configurable directory tree notifications for your monorepo.

A few years ago, we decided to move most of our code into a monorepo. Many advocates have highlighted its upsides, which include better cross-project coordination and simpler dependency management.

But one problem remained: none of the available GitHub integrations for Slack work nicely with monorepos. Slack is vital for day to day communication among Ahrefs’ globally distributed team, so a Slack integration was a must-have feature. We needed a service that could map activity from our various subprojects to their corresponding Slack channels — something existing solutions didn’t offer.

That’s why we built our own integration: Monorobot, a notification bot for monorepos. We’ve improved it iteratively since the monorepo transition, incorporating real time feedback from our engineers over time. Today, Monorobot is an active member of Ahrefs’ Slack workspace, dutifully routing GitHub activity notifications to different channels based on the relevance of each activity.

And now we’re open-sourcing Monorobot, for anybody to use in their monorepo setup! The package is available via OPAM:

opam install monorobot

Read on for more details about the motivation, an overview of the main features, and what’s in the pipeline.

Existing Slack integrations lack monorepo support

Within a monorepo, multiple projects have their code located in separate, nested directories. Correspondingly, each project’s Slack channel is only interested in activity from that part of the overall repository. The issue with most GitHub-to-Slack integrations is that once you subscribe a Slack channel to a GitHub repository, the channel receives all activity from that repository.

Suppose we operate various camel-related services, and we’re planning to launch a new camel ride-sharing app called Camel Ride. The directory structure could look like this:

monorepo/
| frontend/
| | camelride_ui/
| | | mobile/
| | | web/
| | cameldance/
| backend/
| | camelride/
| | | routing/
| | | pricing/
| | camelfood/

As you can see, both the frontend/ and backend/ directories contain code for our fictitious ride-sharing service, along with code from other projects.

If we were to connect the GitHub for Slack integration to this repository, notifications for activity from all projects would be sent to the same channel. Even if I were only interested in activity from the Camel Ride project, I’d need to sift through notifications from the other, unrelated projects. Imagine the volume of notifications this would create for a larger monorepo with dozens of projects. What a mess!

Enter Monorobot

Monorobot, hard at work.

Monorobot enables more granular control over where notifications from the same repository get routed, depending on the type of activity. This routing behavior can be defined in a configuration file named .monorobot.json, which should be committed to the root of the monorepo. Once you create a GitHub webhook from the repository to a running instance of Monorobot, it will use the configuration file to route notifications to relevant channels based on the webhook event payload:

For pushed commits, it checks the path prefixes of the files modified in the commits.
For activity related to PRs and issues, it checks their labels.
For status updates on pushed commits (e.g., CI builds), it uses the same path prefix logic as pushed commits.

Additionally, Monorobot supports unfurling GitHub links shared in Slack.

Path prefix routing for commit push notifications

Continuing with our example, suppose we want to route all commit activity related to the Camel Ride project to a Slack channel called #camelride. Our configuration file might look like this:

{
  ...,
  "prefix_rules": {
    "rules": [
      {
        "match": [
          "frontend/camelride_ui/",
          "backend/camelride/"
        ],
        "ignore": [
          "frontend/camelride_ui/images",
        ],
        "channel": "camelride"
      }
    ]
  }
}

Each rule “matches” a file path to a channel. Whenever someone pushes a commit touching files with either of these prefixes, the #camelride channel will be notified. All other commits will be ignored.

If a file prefix appears in a rule’s optional ignore field, the rule won't be matched even if the prefix is is also in the match field. In the above snippet, the Camel Ride frontend team has decided to silence notifications for activity in the images/ subdirectory.

Now, let’s say the project’s price optimization team is growing, and they’ve decided to create their own separate Slack channel called #camelride-pricing. We can simply commit an update to the .monorobot.json file, and Monorobot will detect the configuration change:

{
  ...,
  "prefix_rules": {
    "rules": [
      {
        "match": [
          "frontend/camelride_ui/",
          "backend/camelride/"
        ],
        "ignore": [
          "frontend/camelride_ui/images",
        ],
        "channel": "camelride"
      },
      {
        "match": [
          "backend/camelride/pricing/"
        ],
        "channel": "camelride-pricing"
      }
    ]
  }
}

Since Monorobot will match the rule with the longest matched prefix, only commits related to the price optimization aspect of Camel Ride will notify #camelride-pricing, and all other general Camel Ride commits will notify #camelride.

There are additional configuration options for prefix rules (and for label rules discussed in the next section) that aren’t mentioned here. Visit the repository for the full details.

Label-based routing for PRs and issue notifications

For activity related to pull requests and issues (opening, closing, merging, commenting, and reviewing), Monorobot uses labels to determine routing. The format is largely the same as for path prefix routing:

{
  ...,
  "label_rules": {
    "default_channel": "notifications",
    "rules": [
      {
        "match": [
          "Camel Ride"
        ],
        "channel": "camelride"
      },
      {
        "match": [
          "Price Optimization"
        ],
        "channel": "camelride-pricing"
      }
    ]
  }
}

Here, all PRs and issues with the “Camel Ride” label will have activity sent to #camelride; those with the “Price Optimization” label to #camelride-pricing; and those with both labels to both channels.

The default_channel field provides an option to fall back on a channel if no rule is matched; this option is available for prefix rules as well.

Status notifications

Monorobot also supports build status notifications for CI pipelines. When it receives a status update for a pushed commit, it routes it to the relevant channel(s) by applying the prefix rules to the commit associated with the build. Further filtering based on status (e.g., ignoring canceled builds, and only notifying for a successful build when preceded by a failed one) is also possible.

Link unfurling

Finally, Monorobot can unfurl links to GitHub repositories shared on Slack (including private ones, if a personal access token is provided). This applies to commit, issue, and pull request URLs.

What’s next

Monorobot is actively used at Ahrefs today, but there are lots of promising future directions it could take. Here, we list a few.

Unifying GitHub and Slack identities

It would be useful to allow GitHub user IDs to be mapped to Slack ones. This would enable more personalized features for Monorobot, such as direct messaging a user when their review is requested or when a CI build fails on a feature branch they authored.

Consolidating notifications

Sometimes, a collection of multiple GitHub webhook events makes sense to be grouped and delivered as a single Slack notification. For example, pull request reviews generate discrete webhook events for each review comment, but it would make more sense to pool them together, so as not to spam a channel with many notifications.

Better status notifications

A CI build failure can have multiple potential causes:

A bad commit
A previous bad commit that has yet to be fixed
An issue with the pipeline itself (this is out of scope for Monorobot)

It can be quite tricky to discern between the first two causes from the GitHub webhook event alone. Cause 1 is handled well by our current approach of using path prefix routing on the commit associated with the build. But with cause 2, that same approach doesn’t always send the build failure notification to the channel where it is actually relevant. In that case, the originator of the initial bad commit won’t be nagged about all subsequent failures, and Slack channels with no relevance to the cause of failure will get polluted with unnecessary notifications.

How can we best determine whether a status notification is “relevant” to a Slack channel? This is still an open question, but one possible direction is to track build state per build step rather than per status, and route notifications based on that. For example, if an overall build fails due to a backend build step failure, then it could be sent to a channel where the frontend team won’t be notified.

Wrapping up

The overall goal of Monorobot is to make Slack notifications more relevant for all teams in a large monorepo environment, using the information available from GitHub webhook events. We’ve had fairly positive results with our own internal usage, and now we hope others find it useful as well.

Monorobot is written in OCaml. We welcome your feedback and contributions on GitHub!

P.S. If anyone does make an actual ride sharing service for camels, do let us know…

Thanks to Feihong, Igor, and Louis for feedback on this post.

Monorobot: a Slack bot for monorepos was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building Ahrefs codebase with Melange

ahrefs — 2021-05-18T15:24:20-00:00

Photo by Jens Lelie on Unsplash

At Ahrefs, we have been using BuckleScript and ReasonML in production for more than two years. We already have a codebase of tens of thousands of lines of code, with several web applications that are data intensive and communicate with backend services written in OCaml, using tools like atd.

Given our investment in these technologies, we have been following closely the recent changes in ReScript, with its rebrand and renaming, and the split with the ReasonML project, explained in the project blog post.

ReScript: becoming its own language

We are excited about the way ReScript is unifying the experience and making it easier for developers who are getting started to find documentation in a single place, as well as continuing its strong focus on performance and readable JavaScript output.

On the other hand, we are trying to figure out the implications of this change in the mid- and long-term, especially regarding the integration with the OCaml ecosystem. And more importantly, what this evolution will mean for production users like us who rely on this integration.

ReScript integration with OCaml has historically been seamless, as BuckleScript started originally as a new backend for the OCaml compiler. However, in recent months, there have been several hints that ReScript wants to evolve towards becoming its own language:

It has now its own parser, incompatible with OCaml native applications
Official repository guidelines for technical writing mentions explicitly that no reference to OCaml should appear in docs
Upgrades to the latest version of OCaml compiler, which used to be part of the roadmap, have been deprioritized recently.

So, even if officially ReScript has not announced that they will break backwards compatibility with OCaml, just the fact that it is sticking with an old version of the OCaml compiler poses some challenges for us in terms of tooling. The uncertainty about the future and the pace of changes add some risk to the high-level goals we have for our teams and codebase: we would like to share more code between frontend and backend, not less.

Melange: a fork of ReScript, focused on OCaml compatibility

When António Monteiro announced Melange, a fork of ReScript but with a strong focus on keeping compatibility with OCaml, we decided to try it out and see how it could work for us.

Ultimately, the experiment was successful. We managed to build all our frontend applications with Melange, while keeping the existing bundling setup, which currently uses Webpack.

Throughout this process, we had to modify some parts of the code. We will now go through the most relevant parts of the process:

Upgrade to OCaml 4.12: the most relevant part was the deprecation of Pervasives module to use Stdlib.
Use ppxlib in our ppxs: we had to upgrade the two ppxs that we use in the frontend codebase to the latest compiler version, bs-emotion-ppx and an in-house ppx for internationalization.
Configure esy: we were already using esy to bring the editor tooling into scope of the developer environment, so we just had to make sure melange would also be included in the json configuration.
Upgrade to Reason 3.7.0: a quite simple change too, as the whole process is automated by using refmt. As a side note, we ran into a small bug with some type annotations, that we were able to work around.
“Lift” dune workspace to the root of our monorepo: this is probably the most intrusive change. Because we have shared code between backend and frontend, and Dune needs to have access to all sources under its workspace, we had to “lift” the Dune workspace from the backend directory to the root of monorepo.

The good

This experiment allowed us to experience what a project like Melange could offer for our use case. Here are some of the things we might be able to leverage in a codebase built with Melange:

Recent version of the OCaml compiler: at some point, we could pin compiler version between backend and frontend teams, making upgrades more straightforward as they would happen atomically.
Shared editor tooling: the official OCaml vscode extension works great with Melange, as well as any other OCaml editor integration. Having backend and frontend teams use similar editor setup removes a lot of maintenance work for us.
Consuming ppxs from source: Melange allows to consume ppxs from source, which also removes issues with pre-compiled ppxs (like this issue with the recent M1 Macs).
Melange allows to run all ppxs from a single executable file, which has some nice performance benefits.
Use Dune for atd files generators: ReScript “generators” are unfortunately not documented anymore, but we use them extensively for atd file generation. Being able to share Dune rules in backend and frontend would make our build setup easier.
Access to OCaml documentation tooling: Melange allows to leverage existing tooling for generating documentation, like odoc.
Async syntax: the latest Reason version supports “let op” syntax, which is handy for client-side code.

The bad

While there are many things that are exciting about Melange, there are some other parts that can be improved.

Build performance: We already knew that performance would be far worse than ReScript, as Melange uses Dune in a way that it was not designed for. In our tests, builds with Melange are roughly 1 order of magnitude slower than ReScript ones.
First-class Dune support: if there was a deeper integration between Dune and Melange, we could explore features like shared libraries or shared rules between backend and frontend. As of today, Dune has no knowledge about Melange environment, so it can perform basic rules execution, but there is no access to high level stanzas like library in Melange.
Two-headed goal: finally, we see a more strategic risk in Melange proposition. Right now it has two goals: keep compatibility with both ReScript and OCaml. But we don’t know how long these goals will be feasible. If at some point ReScript decides to move away from the OCaml compiler fully, then Melange users would not be able to consume any updates to the ReScript ecosystem anymore.

Alright, but are you migrating to Melange or ReScript?

With all the information available, the answer is: we don’t know yet. 😄 We want to keep exploring all the available options and have as much information as possible before committing further. So for now, we are upgrading the codebase to recent versions of ReScript, but we are holding up on features that only work one way. For example, we have not migrated our codebase to the ReScript syntax yet, as there is no way to translate back to Reason syntax.

In the meantime, we will keep exploring how far the limitations of Melange can be mitigated. To be continued! 🚀

Thanks to Igor and Feihong for reviewing and improving earlier versions of this post.

Building Ahrefs codebase with Melange was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

One and a half years of ReasonML in production

ahrefs — 2020-07-26T15:19:31-00:00

Photo by https://unsplash.com/@willianjusten

The first Reason application at Ahrefs went online on January 31, 2019. Since then, many more applications have been either rewritten in Reason, are being slowly migrated from React to ReasonReact, or are conceived from the start as Reason projects. It is safe to say that the bet placed on Reason paid off big time. We will never go back to doing pure JavaScript again, with the possible exception of simple backend scripts.

In the past few years, it’s come to light that there are a number of other large Reason/BuckleScript codebases in the wild, but there still isn’t a ton of information out there about what it’s really like to work with Reason in production. To help remedy that, we thought it would be instructive to ask each of our frontend team members what their Reason journey has been like so far.

We gave them the following questions as starting points (but they were free to talk about anything they wanted):

How does Reason compare to other languages you’ve used in the past?
What’s your favorite thing about Reason?
What’s your least favorite thing about Reason?
How does ReasonReact compare to other frameworks you’ve used?
Was it easy to pick up Reason? Why or why not?

Javi

How does Reason compare to other languages you’ve used in the past?

In the past I worked with languages like Java, C, or less known like Pascal or Prolog. But the languages I’ve spent more time with are Objective-C and JavaScript. The main difference between all those languages and Reason is the exhaustiveness that you get from OCaml type checker. This is maybe awkward, but it feels like you stop coding alone and suddenly you have a sidekick always sitting next to you, that is helping you notice the things you forgot about, or found new code that is not consistent with code you or someone else wrote before.

In a world that is moving towards remote work, where many of us spend hours every day coding physically far from our colleagues, it makes the experience much more delightful. Plus, it allows for teams working on different time zones to keep a healthier work-life balance, because there is less need to have synchronous communication than with more dynamic languages, as more assumptions and design decisions are “embedded” into the code.

What’s your favorite thing about Reason?

Can I pick two things? It’s hard to choose only one.

The first one is the exhaustiveness and quality of the type checker, as mentioned above. Sometimes it takes a bit longer to build a feature than what it would in other languages, until the types are figured out. But this is largely compensated by the confidence one has when shipping code to production, or diving into large refactors.

The second one is the speed of the BuckleScript build system, which is built on top of ninja. I had never worked with such fast build system. As an example, we have recently started to use remote machines to develop at Ahrefs. In one of these machines that has 72 cores, BuckleScript takes roughly 3 seconds to clean build all our Reason code: application, libs, decoders… everything. Many tens of thousand lines of code! We thought there were something wrong, but we realized the compiler is just So Blazing Fast™️.

What’s your least favorite thing about Reason?

I guess we’re going through a necessary stage until things stabilize in the future, but there is a lot of fragmentation at the moment between “Reason native”, which tries to stay closer to OCaml, and “Reason web”, which has a goal to become friendlier for JavaScript developers.

I am excited to see what BuckleScript new syntax will lead to, but I would also love to see a “universal” solution that works for the main use cases out of the box, becoming sort of Rails for Ocaml or Reason. sihl is a project that seems to go in that direction and looks very promising.

How does ReasonReact compare to other frameworks you’ve used?

I consider ReasonReact mostly like React + types on top, because the bindings layer is very thin. The thing that I like most about React is that it follows the Unix philosophy: it does one thing and it does it really well. Maybe we have forgotten already today, but having to maintain and mutate UI based on data updates was one of the main sources of bugs in the past. The other nice thing is that there is so much good content about it: blog posts, documentation, etc.

Was it easy to pick up Reason? Why or why not?

It took some time, as with any other language. We have things like syntax or semantics much more ingrained into our brains than we think, so there is always some “rewiring” time that is needed to learn a new language, even if Reason makes an effort to stay close to JavaScript syntax. The most challenging part was probably the bindings one, because coming from JavaScript, there are no previous knowledge that one can use as foundation to build upon, it’s all “new knowledge”. glennsl BuckleScript ffi cheatsheet was a huge help for me.

Ze

I really like working with Reason, and have wanted to do so for a while. I was quite happy to see that working with it matched my expectations.

You get so much support from the type system, and still have a lot of flexibility to represent your domain model. Coming from other languages or paradigms, you don’t feel limited at all in what you can achieve.

The language has such a strong type system that you feel much more comfortable with your coding.

The OCaml type system is there to make sure you code with assurance. This is especially true when refactoring code. You can be sure that everything will work fine after it compiles. If it compiles, it works :)

It’s also very helpful when working on a monorepo. You don’t have to keep reading the source code of everything you use to make sure you don’t have types mistakes. Changes in code in one lib reflect immediately in all the others. This makes the feedback loop much shorter and safer.

The editors integrations with the type system are quite good and help a lot to write code better and faster.

Also, compilation times are super fast.

Last, but not least, ReasonReact is, for me, the hidden gem of ReasonML. The newcomers that have some difficulty with the language should start with it. IMHO, ReasonReact is simpler and has a better developer experience than React itself. It should be the gateway drug frontend developers need to get started with Reason/OCaml 😄

Liubomyr

To me, all those language features boil down to one essential thing, and it’s the easiness of refactoring. New business requirements popups all the time, and often your initial code assumptions are no longer correct. It was such a pain to modify code in a large JS codebase, as you never know how many things you potentially break in the process. With Reason, it has never been easier. If you need to change your data shape or some component API, you just do it, and from there, the compiler will guide you through all the places you broke, and help to fix those.

Coming from the JS world, it feels like the initial development is slower, because of the learning curve, missing bindings, less StackOverflow answers, but in the end, you are getting a stable software which is way easier to maintain and add features to.

Egor

I switched to Reason when I joined Ahrefs team about a year ago, before that I worked mostly with Ruby language.

The first thing that impressed me in ReasonML was code refactoring. Refactoring in language with a strong type system, like ReasonML and OCaml, is much easier than what I am used to. If your program compiles after your refactoring — most likely you did everything right, if it doesn’t compile — you can immediately see what you forgot to change. This can be achieved in languages with a dynamic type system only with a huge amount of code tests (supporting big test suite is a time consuming process as well as code support).

The other thing that I really like about ReasonML codebase — how readable it is. When you just enter into ReasonML world — some things can be unfriendly from the first sight, for example, immutable let bindings, but in the end, you realize that these language decisions help you to write cleaner and simpler code.

Seif

The programming language I used the most in the past is JavaScript. I switched to Reason when I joined Ahrefs a few months ago. From the start, I worked mainly on the code shared by the majority of the tools and I don’t think I would have had the same confidence making changes if I was doing it with JavaScript. I love JavaScript’s developer experience and accessibility. Reason provided me predictability without hurting these very same things I like about JavaScript.

Bryan

Reason (and OCaml) is, by far, one of the easiest languages to work with. Easy in the sense that the compiler helps eliminate an entire class of errors so you don’t have to worry about them. Additionally, in most other web-centric languages, it’s a pain to add features to existing code that you’ve not touched for a long time. With strong static typing, I can usually add the feature I want in either the backend or frontend, and then let the compiler tell me what needs to be updated.

Pattern-matching is one of my favourite features in Reason. To me, it makes more sense to be able to explicitly specify conditions that I’m interested in a clear and concise manner, and let the compiler tell me if I missed out a particular condition. Records go hand-in-hand with this. As software programs are made up of data and instructions, records are the perfect data containers. They are quick to define and query, focusing on data rather than behaviour (think classes and instance methods).

It definitely took a while to pick up Reason mainly because it takes time to become familiar with idiomatic OCaml. But once I crested that learning curve, everything just made sense and all the features of the language that made Reason seemingly difficult to learn — strong typing, the functional paradigm, etc, became assistants that helped me to write better code.

Feihong

ReasonReact is a great library for making complex UIs in a large codebase because you get the familiarity of React coupled with the type safety of OCaml. Having two well-established technologies in its foundation is a big advantage that ReasonReact has over other functional UI libraries/frameworks in the transpile-to-JS universe. I didn’t have any professional OCaml experience before joining, yet the ramp up was made much easier by my existing knowledge of React and the (somewhat superficial) similarity of the Reason syntax to JS. Oftentimes it was possible to correctly guess the intent of existing Reason code without knowing all the syntax, because most React concepts carry over pretty directly. And even though the documentation is incomplete and not perfect, it’s quite usable already and among conceptually-similar frameworks is second only to the Elm documentation.

The compiler errors were difficult to get used to at first. The compiler is fairly good at pointing out the location of the error, but not necessarily as good at explaining the nature or cause of the error. As such, having a REPL would be extremely useful. Actually, OCaml does have its own REPL, but BuckleScript (the compiler used by Reason to translate OCaml to JS) does not at the moment. Nonetheless, the Try Reason page is a really good tool to try out small snippets of code and is extremely useful while learning the language (we will still occasionally post Try Reason links in our slack channel).

Summary

The reality is that Ahrefs has always been an OCaml shop, but in the past OCaml was only used to build the backend. Now that we are also using it on the frontend, we get the benefits that our backend colleagues have enjoyed for many years: the expressiveness afforded by pattern matching, the ease of refactoring in large codebases, the stability of a mature programming language, and the confidence of “if it compiles, it works”. To make a shoddy nautical analogy, it is as if we had built a wooden ship powered by a turbo engine. But now the wooden parts are being replaced with steel and plastic, bringing the exterior of the ship up to modern standards as well. As a result, the ship runs faster and more reliably, making the passengers (our users) more satisfied. Also, pirates (bugs) have a harder time hijacking the ship because it’s sturdier and defended by well-disciplined camels. Because the ship keeps getting more and more passengers who want to experience a delightful ride and take pictures with enigmatic camels, we require a constant influx of willing and able boat engineers (who aren’t allergic to camels) to extend and maintain the ship. (Yes, that means that we are hiring️.)

Thanks to Raman and Louis for fact checking this post.

One and a half years of ReasonML in production was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to write a library for BuckleScript and Native

ahrefs — 2019-10-22T10:09:09-00:00

Written with Javier Chávarri and Feihong Hsu.

The first Reason Conf US just ended. Many talks mentioned native compilation. Sharing code between BuckleScript and native artifacts is a use case which is more and more common. This blog post is an introduction on how to set up a library available for both worlds, sharing as much code as possible.

The goal

What we try to produce is a library with an identical interface for BuckleScript and native. But without duplicating code. It should also be possible to have some parts of the library that are a different implementation depending on the target, as we want to be able to leverage existing libraries that are working only in one of the worlds.

The build systems

For BuckleScript, there is only one build system: bsb. It is driven by a bsconfig.json file. And is installed as part of the bs-platform.

On the native side, there are a lot of different build systems that are available. But recently one of them became a de facto standard: dune. It works with a very minimal amount of configuration. And it supports the reason syntax by default.

These two tools are working in a way which is pretty similar. They share a lot of concepts. And it is easy to set them up so that both are working in the same codebase.

The main similarities that interest us are:

The ability to work on specific source directories
Namespacing in bsb and wrapping in dune are both putting all the
files of the library under a single module name

The source code file tree

The code of the library is split into 3 directories.

├── js/
├── native/
└── shared/

shared is meant to host most of the code and all the code in this directory will be compiled in both modes.
js contains the parts that are specific to BuckleScript.
native contains the parts that are specific to native OCaml.

Set up the build systems

Once we have our basic skeleton for the library, it is time to set up the build systems. We want to have two configurations as similar as possible to make them easier to understand. Once we are done, the tree will look like this:

├── bsconfig.json
├── dune
├── dune-project
├── js/
├── native/
└── shared/

BuckleScript

At the root of the library we need a bsconfig.json file to drive
bsb. The documentation is available at https://bucklescript.github.io/docs/en/build-configuration.

The main part for us is sources. We will use it to tell bsb to look at the js and shared folders. We also want to set namespace to true, which will wrap all your project’s files under a common module name.

  "namespace": true,
  "sources": [
    {
      "dir": "js",
      "subdirs": true
    }, {
      "dir": "shared",
      "subdirs": true
    }
  ],

The rest of the file is as usual.

{
  "name": "sharedlib",
  "namespace": true,
  "sources": [
    {
      "dir": "js",
      "subdirs": true
    }, {
      "dir": "shared",
      "subdirs": true
    }
  ],
  "package-specs": {
    "module": "es6",
    "in-source": true
  },
  "refmt": 3,
  "suffix": ".bs.js",
  "generate-merlin": true,
}

Dune

We must also add a dune file to the root of the library. For dune, we have different options — it is possible to ignore the js directory but read everything else. Or to check only shared and native. To make the configuration similar to BuckleScript, we will go with the second solution.

The dune directive to do that is dirs. By defaults it tells dune to explore every directory except the ones hidden (starting with a dot) or starting with an underscore. More details in dune’s documentation. To make it do what we want, the configuration should be:

(dirs shared native)

We also use another option of dune to tell it to include the content of those two directories as if it was at the root of the project. Without this stanza, dune would only use the source files at the root of the project and ignore everything in the sub directories.

(include_subdirs unqualified)

Then we need the usual library stanza to give a name to our library, state the dependencies, compilation flags, etc. In our simple case, the only information needed is the name. We can explicitly set wrapped to true, but this is already the default behavior. The documentation for the whole library stanza describes how to specify more details.

The final dune file looks like this:

(dirs shared native)
 (include_subdirs unqualified)
 (library
  (name sharedlib))

We also want a basic dune-project. If we don’t write it by hand, dune will generate it for us. I am using version 1.10 as an example. But it can be changed to whatever version suits your project.

(lang dune 1.10)

Compilation

With the setup described above, the compilation for BuckleScript and native is the same as in a setup with only one or the other.

bsb -make-world for BuckleScript
dune build @all for dune

The call to bsb is usally put in package.json in the scripts part, so that the usual yarn build can be used. For native, it depends if you rely on esy or opam.

How to consume the library

This is exactly the same setup that would be used in a pure BuckleScript or pure native library.

To use your library in BuckleScript:

Add the name and version to package.json
Add the name to bsbconfig.json of consuming library/app

To use your library in native OCaml, add the name of your library to the libraries part an executable or library stanza, e.g.

(executable
 (name main)
 (libraries sharedlib))

Module naming

If you want your module name to contain capital letters in the middle (e.g. TeenageMutantNinjaTurtles), then be aware that name munging works differently between bsbconfig.json and dune. For example, if you want to refer to your module as CoolSharedLib in your code, then the name in bsbconfig.json must be cool-shared-lib, and in dune it must be coolSharedLib.

Platform specific code

The whole library does not have to be exactly the same in the two platform. It is possible to add modules that are available only in one mode. Or to have modules with a different interface.

For example, by adding a file Foo.re in js but not in native, the library now has a module Foo available when compiled to javascript. But only when compiled to javascript.

Downsides

Both bsb and dune generate .merlin files when they compile our library. They override each other. It might be troublesome if the version of ocaml used for native code is not 4.02.3. Simply recompile the library for your platform to solve the problem.
Out of the box, this approach doesn’t really allow us to share interface files between both platforms: native and BuckleScript. One workaround for that, if we wanted to share some module Foo, is to:
1. add Foo.mli or Foo.rei file in shared
2. add include FooImplementation in Foo.ml
3. add FooImplementation in both native and js folder
It’s not possible to be platform specific for just a few lines of code (e.g. if IS_NATIVE foo else bar), the minimal per-platform unit is a file/module.

Example project

We have set up a simple library to showcase what a repository looks like once the whole configuration is in place. It is available on github.

For now the repository contains only a library. But with this setup, it is actually possible to build an executable too. It is also possible to enrich it, for example by adding atdgen to communicate between both sides of the library.

How to write a library for BuckleScript and Native was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Getting started with atdgen and bucklescript

ahrefs — 2018-09-12T02:53:58-00:00

atdgen is a project to create types and data structures that can be serialized to JSON. It is very convenient when communicating between multiple processes, creating a REST API or consuming JSON objects from other tools. It can be compared to JSON schema or Protocol Buffers, but with richer types and more features.

The idea is to write a list of types in a specification file, an .atd file. Then running atdgen, it is possible to generate OCaml or Java code to serialize/deserialize values of those types to/from corresponding json.

Until very recently, atdgen could generate code only for native OCaml. But the support of bucklescript has been merged! atdgen the cli tool is still a native OCaml binary. But it can output some OCaml code that can be compiled using bucklescript.

The work to implement this new feature of atdgen has been funded by Ahrefs. We highly appreciate open source tools. And as much as possible, we prefer to contribute to existing open source projects rather than to re-invent the wheel internally.

Installation

To install atdgen we first need to install opam (OCaml package manager), as atdgen doesn’t provide ready to use binaries and is only distributed as source package via opam. The procedure is simple and documented here: https://opam.ocaml.org/doc/2.0/Install.html

Then we need to initialize opam and create a switch. Any version of ocaml greater or equal to 4.03.0 should be fine.

opam init -a
opam switch create . 4.07.1 -y

Once it is done, we have to install the development version of atdgen. The support of bucklescript is not officially released.

opam pin add atd --dev-repo   
opam pin add atdgen --dev-repo

Make sure that atdgen is available.

$ which atdgen                 
(current $PWD)/_opam/bin/atdgen

Of course, we need bucklescript.

yarn init                 
yarn add bs-platform --dev

We also need the bucklescript runtime for atdgen, as it is not currently provided by atdgen itself. So we have written and open-sourced our version of the runtime : https://github.com/ahrefs/bs-atdgen-codec-runtime.

This runtime is responsible for the conversion between JSON values and OCaml values. The JSON values are based on the standard Js.Json.t type provided by bucklescript to be sure that it is easy to interoperate with the rest of the ecosystem.

It is published on npm for easy integration in bucklescript projects.

yarn add @ahrefs/bs-atdgen-codec-runtime

Project configuration

After the previous section, package.json should be almost ready. We can add a few scripts to make it more convenient to compile the project. Here is how it should look once completed.

{
  "name": "demo-bs-atdgen",
  "version": "0.0.1",
  "description": "demo of atdgen with bucklescript",
  "scripts": {
    "clean": "bsb -clean-world",
    "build": "bsb -make-world",
    "watch": "bsb -make-world -w",
    "atdgen": "atdgen -t meetup.atd && atdgen -bs meetup.atd"
  },
  "devDependencies": {
    "bs-platform": "^4.0.5"
  },
  "peerDependencies": {
    "bs-platform": "^4.0.5"
  },
  "dependencies": {
    "@ahrefs/bs-atdgen-codec-runtime": "^1.0.4"
  }
}

The bucklescript configuration is very simple. We use the basic configuration that can be found in any bucklescript project. Except that we need to add one dependency to bsconfig.json:

{
  "name": "demo-bs-atdgen",
  "version": "0.0.1",
  "sources": {
    "dir": "src",
    "subdirs": true
  },
  "package-specs": {
    "module": "commonjs",
    "in-source": true
  },
  "suffix": ".bs.js",
  "bs-dependencies": [
    "@ahrefs/bs-atdgen-codec-runtime"
  ],
  "warnings": {
    "error": "+101"
  },
  "generate-merlin": true,
  "namespace": true,
  "refmt": 3
}

First ATD definitions

It is time to create a first .atd file, containing our types. This part is also documented on https://atd.readthedocs.io/en/latest/tutorial.html#getting-started

For this example, I decided to go with a meetup event. Put the type definitions in src/meetup.atd.

(* This is a comment. Same syntax as in ocaml. *)

type access = [ Private | Public ]

(* the date will be a float in the json and a Js.Date.t in ocaml *)
type date = float wrap <ocaml module="Js.Date" wrap="Js.Date.fromFloat" unwrap="Js.Date.valueOf">

(* Some people don't want to provide a phone number, make it optional *)
type person = {
  name: string;
  email: string;
  ?phone: string nullable;
}

type event = {
  access: access;
  name: string;
  host: person;
  date: date;
  guests: person list;
}

type events = event list

We use the atdgen binary (compiled previously) to generate the ocaml types and the code to serialize/deserialize those types.

atdgen -t meetup.atd # generates an ocaml file containing the types
atdgen -bs meetup.atd # generates the code to (de)serialize

The generated files are:

meetup_t.ml(i) which contain the ocaml types corresponding to our ATD definitions.
meetup_bs.ml(i) which contain the ocaml code to transform from and to json values.

At this point we can compile our project.

yarn build

If everything worked properly, we now have two .bs.js files in the src directory.

$ tree src
src
├── meetup.atd
├── meetup_bs.bs.js
├── meetup_bs.ml
├── meetup_bs.mli
├── meetup_t.bs.js
├── meetup_t.ml
└── meetup_t.mli

0 directories, 7 files

At this point, we can create new OCaml/Reason files in the src directory and use all the code atdgen generated for us. Two examples to illustrate that.

Query a REST API

A common usage of atdgen is to decode the JSON returned by a REST API. Here is a short example, using the reason syntax and bs-fetch.

let get = (url, decode) =>
  Js.Promise.(
    Fetch.fetchWithInit(
      url,
      Fetch.RequestInit.make(~method_=Get, ()),
    )
    |> then_(Fetch.Response.json)
    |> then_(json => json |> decode |> resolve)
  );

let v: Meetup_t.events =
  get(
    "http://localhost:8000/events",
    Atdgen_codec_runtime.Decode.decode(Meetup_bs.read_events),
  );

Read and write a JSON file

Atdgen for bucklescript doesn’t take care of converting a string to a JSON object. Which allows us to use the performant json parser included in nodejs or the browser.

let read_events filename =
  (* Read and parse the json file from disk, this doesn't involve atdgen. *)
  let json =
    Node_fs.readFileAsUtf8Sync filename
    |> Js.Json.parseExn
  in
  (* Turn it into a proper record. The annotation is of course optional. *)
  let events: Meetup_t.events =
    Atdgen_codec_runtime.Decode.decode Meetup_bs.read_events json
  in
  events

The reverse operation, converting a record to a JSON object and writing it in a file is also straightforward.

let write_events filename events =
  Atdgen_codec_runtime.Encode.encode Meetup_bs.write_events events (* turn a list of records into json *)
  |. Js.Json.stringifyWithSpace 2   (* convert the json to a pretty string *)
  |> Node_fs.writeFileAsUtf8Sync filename  (* write the json in our file *)

Full example

Now that we have our functions to read and write events, we can build a small cli to pretty print the list of events and add new events.

The source code of the full example is available on github.

You can run it like this:

$ echo "[]" > events.json
$ nodejs src/cli.bs.js add louis louis@nospam.com
$ nodejs src/cli.bs.js add bob bob@nospam.com
$ nodejs src/cli.bs.js print
=== OCaml/Reason Meetup! summary ===
date: Tue, 11 Sep 2018 15:04:16 GMT
access: public
host: bob <bob@nospam.com>
guests: 1
=== OCaml/Reason Meetup! summary ===
date: Tue, 11 Sep 2018 15:04:13 GMT
access: public
host: louis <louis@nospam.com>
guests: 1
$ cat events.json
[
  {
    "guests": [
      {
        "email": "bob@nospam.com",
        "name": "bob"
      }
    ],
    "date": 1536678256177,
    "host": {
      "email": "bob@nospam.com",
      "name": "bob"
    },
    "name": "OCaml/Reason Meetup!",
    "access": "Public"
  },
  {
    "guests": [
      {
        "email": "louis@nospam.com",
        "name": "louis"
      }
    ],
    "date": 1536678253790,
    "host": {
      "email": "louis@nospam.com",
      "name": "louis"
    },
    "name": "OCaml/Reason Meetup!",
    "access": "Public"
  }
]

Getting started with atdgen and bucklescript was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Skylake bug: a detective story

ahrefs — 2017-06-28T18:34:51-00:00

It was a dark and stormy night; the skylake CPU buzzed with excitement, and then, suddenly, the hyperthreads started to lock up..

Or something like that.

This week a new erratum for the Intel Skylake and Kabylake processors families was brought to public attention on the Debian mailing list, and then on various social media and news outlets.

We have been investigating this issue since January with the core OCaml team, as we were struggling with a mysterious bug affecting our developers machines, and ultimately our production system, resulting in a corruption of important data in our databases.

At Ahrefs, we operate a fleet of thousands of servers, running a wide variety of services (huge web crawler among others). At this scale, dealing with unexpected application behaviors is common. While we try to reduce the probability of the software not functioning as expected, bugs are sadly a real part of our everyday life. Even though we can assume the underlying hardware running any infrastructure can be thought of as more reliable and less prone to bugs than software components, issues can still arise in unexpected ways. When the number of servers increases, it is not unusual to observe faults in the hardware preventing the system from functioning as specified.

It is certainly not frequent to encounter such problems in CPUs but reading through the list of errata published by any manufacturer, each CPU model contains a fair amount of bugs. This story is about the bug in the microcode of Skylake processor leading to incorrect code execution under certain conditions. This is certainly scary at first sight: how can we trust our system if we cannot trust its main component ? Yet, like software bugs, processor defects can be identified, contained, and we can take actions to prevent them from impacting the operation of the infrastructure.

We do not know the full implications of this particular bug, especially security implications in case of untrusted code execution. But we’d like to tell the story of this erratum from our point of view, to provide some context, and show that dealing with it was not much different than dealing with any usual software flaw. While this post aims to cover our own perspective on this adventure, we would like to thank Mark Shinwell, Xavier Leroy, Frédéric Bour, everyone involved in the Mantis issue and the OCaml IRC channel for their help and time spent investigating with us. Update: Xavier Leroy told his own side of the story in another blog post.

Setting the scene

Our story starts in late 2016 after some of our backend developers received new laptops to work on. After a few days Enguerrand Decorne noticed unusual crashes during compilation of our OCaml codebase.

This issue, considered mildly annoying at first, seemed to affect only Enguerrand’s machine. For a few days no other machine would exhibit the same behavior, so we figured this was a fault specific to his system configuration.

However, concerns were subsequently raised after witnessing the generation of invalid machine code and later on, after the deployment of a service on one of our new clusters composed of Skylake Xeon processors, leading to the insertion of corrupted data into our storage system. The priority raised from the annoying level, to potentially critical. Other developers started working together to obtain more information and assess the impact on our infrastructure. Soon after we were able to reproduce the issue on several machines.

The remainder of this post is a technical description of the steps taken to ensure that our systems were operating safely. It is intended to show that such low level CPU issues is not necessarily fatal — in less than two weeks, with the great help of core OCaml developers, we identified the conditions of the crash and set up a workaround.

Tracking down crashes in OCaml

Most of our backend code is written in OCaml, a high level and expressive language supporting functional programming style (among others), which allows us to develop robust systems with ease, thanks to its strong type system and mature legacy.

The compiler segfaults were definitely a surprise, since this shouldn’t happen for any program written in OCaml, as type system and other features (such as automatic bounds-checking) usually guard us from such errors. However, stack overflows can be possible sources of segfault (when a non-optimal recursion is running too deep), so our first intuition was to increase the stack size when running the compiler. This didn’t change anything, and the reported fault address wasn’t anywhere near the stack address bound.

Before witnessing the crash on other machines, we suspected a failure in the virtualization software used by our two developers that were able to reproduce the crash, who use VMware as a part of their development workflow. We tried early on to switch to Virtualbox, but the migration proved itself fruitless as the crashes kept appearing. After a short while we began encountering the same issue on physical machines, so we ruled out a possible virtualization software bug.

The usual debugging process for crashing OCaml code didn’t prove effective — we needed to narrow down our approach.

OCaml ships with two backend implementations: a bytecode interpreter and a native compiler. We were able to reproduce the issue using both a native compiler and a compiler running on the bytecode interpreter. Consequently, this ruled out a miscompilation coming from the code emitted by the compiler, the OCaml runtime itself was misbehaving.

The runtime code is written in C, and implements low level functionalities, including the garbage collector used by both backends. After rebuilding the runtime with debug symbols, we were able to retrieve a proper stack trace and core dump. The stack trace pointed to the garbage collector’s mark phase. OCaml’s GC is a classic generational mark and sweep collector. The mark phase walks the heap starting from pointers on the stack and other registered root values, and marks every reachable block of memory.

Further inspection with gdb of the frame and address of crash revealed that the marking code encountered a corrupted block header with invalid size information, causing what looked like a buffer overrun error. Each memory block allocated in OCaml heap begins with a header word, storing metadata used by the GC, including a tag describing the kind of value present in this memory block. The header contains the size of the block, and the crash happened when the mark code was attempting to scan an array which was supposed to be more than 1TB large.

This was obviously not the cause of the problem but rather the consequence: something corrupted the header word after this block had been properly allocated, postponing the crash until the next GC cycle. It was the right time to escalate the issue to the OCaml bugtracker, after isolating a proper test case to reproduce the issue.

A set of strange leads

Escalating the issue to Mantis made us to take a step back and gather our findings, and we quickly got great feedback from the OCaml core team.

At this point, what does the problem look like?

We only had sparse information, but dmesg gave us interesting data point. When a page fault occurs and the kernel detects an incorrect memory access, it logs a line in kernel log buffer containing the fault address, the instruction pointer and stack pointer.

[22985.879907] ocamlopt.opt[48221]: segfault at af8 ip 00005564455169bd sp 00007ffc9f36b130 error 4 in ocamlopt.opt[556445006000+613000]

Next to the 3 addresses, already available in the coredumps, an error code is reported. This number in decimal form is actually a bitset, and the flags are documented in the Linux kernel sources in arch/x86/mm/fault.c. Error 4 can thus be read as a read access page fault from user mode, trying to read memory which had not been previously mmap’ed.

Error codes reported following our crashes involved protection faults or access to unmapped addresses, which corroborated our earlier buffer overrun hypothesis. More interestingly we witnessed a crash with the PF_RSVD flag enabled. This left us puzzled, none of us had ever seen such fault before. Apparently it indicates that the the page table was somehow corrupted, with some entries having non-zero bits reserved by the x86 architecture specification.

It was scary that the corruption would escape the process address space, and to our limited knowledge, it could only have been caused by kernel issue or potentially hardware issues, like memory errors. Yet we were able to reproduce this on several machines with different kernel version, and different hardware. We blamed virtual machines earlier but this theory was debunked already. We still have no explanation at this time, and pursuit on this front would require intimate knowledge of virtual memory implementations that we didn’t have.

One developer wasn’t able to reproduce the problem at all on his machine after hours of testing, but something was fishy: it didn’t sound right that an OCaml runtime bug would be able to modify the page table. Maybe it was some corner case with reserved addresses, but this something was beyond our reach here. Out of ideas, it was time to get some assistance from tools intended to track memory corruptions, like asan and ubsan.

Running Asan didn’t yield any meaningful results. Valgrind was later tried, following advises from the OCaml team, but every tools were preventing the crash. Quickly reproducing the bug for testing required running code in a loop, keeping the CPU and memory fully busy.

This was harder to do on developers machines, due to limited resources and other processes running, and Address Sanitizer would only increase the resources usage. Dedicating a powerful server would make further investigations more comfortable, and increase the likeliness of reproducing with instrumented code.

But with great surprise, it was not possible to reproduce the problem on a server machine, with and without instrumented code. This is when we realised that all the machines exhibiting the crashes were running a processor of the Intel Skylake processors family, while the server and other developer machines had CPUs from the Broadwell family.

The hardware, an unusual suspect

In the meantime several core OCaml developers had been closely investigating the issue and started auditing recent changes in the runtime, and identified a few suspicious changes and known bugs.

Certainly they were more qualified for this task, but it acted as an incentive to examine the history of this bug from our angle. At first, we had assumed that the bug was specific to the new laptop with virtual machines. This could not explain why the crash never manifested on older workstations equipped with Skylake processors. Several other developers had been using them for a few months, and only noticed the crash after awareness of the issue had been raised by Enguerrand.

What had changed, besides Skylake? Only a few week before, an internal migration from OCaml version 4.02.3 to 4.03.0 was rolled out in our codebase. Intrigued, we went ahead and tested OCaml 4.02.3 again, which showed no memory corruptions after several tests. It was time to browse the OCaml changelog for runtime related entries. The search stopped quickly on a promising item in the list: the OCaml C runtime build optimisation level had been increased to -O2 from -O1.

Could the optimizations dig out an undefined behavior in C code, leading to bad assumptions in the GC code corrupting the heap ? Rebuilding the runtime with -O1did not corrupt memory, so the source of the corruption was in the runtime and was triggered by some gcc specific optimization pass. This sounded like undefined behavior, although the information we had led us to some hardware bug.

The next day, Xavier Leroy commented on the bugreport reporting that the crash had been observed in the past. Another industrial OCaml user was affected, and they had discovered HyperThreading was part of the necessary conditions. After running the test case for several hours on several machines with HT disabled in the UEFI setup, it was clear we were facing a similar situation. This led to the hypothesis of a hardware bug:

Is it crazy to imagine that gcc -O2 on the OCaml 4.03 runtime produces a specific instruction sequence that causes hardware issues in (some steppings of) Skylake processors with hyperthreading? Perhaps it is crazy.

This possibility had struck us too, motivated by the HyperThreading, the page table corruption and the Skylake specific set of conditions.

This issue had certainly a strange profile. But nobody was ready to fully embrace the cpu bug hypothesis yet. We convinced ourselves that disabling HT could affect cache pressure and unfold some undefined behaviours.

HT could also explain the non-determinism, since cache pressure would depend on timings and scheduling. None of us had sufficient experience in this area to assess the strength of such hypothesis, and we did not quite buy it on a single threaded OCaml program. Our debugging motto claims that “assumptions are not facts”.

It was time to browse Intel errata list and attempt to update the CPU microcode. Although, the errata descriptions are formulated in vague terms, none of the issues disclosed at this time were looking similar to the situation under investigation. Unfortunately, CPUs microcode had no fix waiting for us either. OCaml developers investigated the errata list from their side but the lack of detailed information turned this into a fruitless and complex task.

In the absence of better alternative, we focused our work on pinpointing the exact source of the crash as if it was a software bug, in the hope of either finding a code issue or ruling out this hypothesis while getting more detailed data. We needed a way to identify the problematic code and find a workaround. From our side, it was not only a matter of finding whether or not there was a bug in OCaml code, but more crucially we needed a guarantee on the quality of our generated code running critical services in production.

Identifying the offending code

The other OCaml user affected by this issue reported that they had solved the problem by switching to another C compiler. Building the runtime with clang instead of GCC would prevent the GC from crashing. They also suggested to obtain a diff of the generated assembly. Indeed, once built with clang, the runtime would not crash. But clang generates widely different assembly from GCC and we did not have the resources to analyse several hundred thousand lines of changes.

If we could isolate the problematic C code, comparing the generated code would be easier. The problem had the form of a well known nail:

Around 50 C files composing the OCaml runtime,
There is a good state (when built with gcc -O1)
And a bad state (when built with gcc -O2)

This nail comes with a precious hammer: bisection.

The bisection approach had a downside in this occasion. Any state can be labeled bad with certainty as soon as the test crashes, but we would need to wait several hours to be confident enough to trust a non crashing test as good data-point. The reproducibility was not always consistent and a non-crashing state could be a false negative still waiting to trigger the conditions leading to the crash. A reduction of search space was necessary.

All the coredumps we had showed that the fault was caused by a corrupted heap block header, and our testcase involved the compiler. The OCaml compiler is not 100% deterministic, and IO/s primitives and unix environment in the runtime can affect timings and allocation patterns. But it sounded sensible to assume that the code corrupting a heap header block was also the code reading and writing those blocks: the major GC.

This hypothesis made bisecting fast: the first file we tried, major_gc.c, turned out to be the one. To make sure it was not a subtle issue in linker, reordering symbols or code blocks, we tried a few others files and confirmed changing the optimization level of some other files alone made no difference.

But the generated code difference was still way too large. Bringing this topic up on the OCaml IRC discussion channel led to some useful inputs. We were taught that gcc supports an attribute to enable specific optimizations at the function level, using __attribute__((optimize("options,..."))). Following the same strategy, it was easy to trace the source of the malfunctioning code to the sweep_slice function, which implements the sweeping phase of the classic mark and sweep garbage collector for the old generation.

Ignoring the subtle details of incremental GC, the sweep_slice function is the last pass of a normal major collection cycle. It is responsible for scanning all blocks in the major heap, and reclaiming unreachable blocks to the list of unallocated space.

The bulk of this function is a switch taking action for each block depending on its status :

https://medium.com/media/cfbc19fddccc5f2c1a76fcc802fae049/href

This finding felt consistent with the information at hand. When the block is reachable, the color (describing the reachability status of the block) is reset. If the block became unreachable (while color) - it is reclaimed. In both cases, the block header is modified.

Getting back to the assembly diff.

Nobody in the team knows a great deal about assembly and we only have a really basic understanding of most of the instructions used in both versions. It quickly became obvious that the noise level in this diff, with thousands of lines of changed, was still too high for us to spot anything related to the problem. This problem was getting far beyond the common knowledge of everyone in the team.

But this was still sounding like your day to day bug tracking process. The less you know, the more careful you need to be, tackling the problem step by step. We stuck to what approach had served us well until now: bisecting.

We went through the list of optimisation passes enabled by GCC at -O2. This is a fair amount of optimisation passes and it would have been too time consuming to try them one by one, given the time needed to trigger the crash. Yet we had a hint: a memory corruption was happening semi randomly in the garbage collector. We were also keeping the undefined behaviour bug as a potential explanation. It was likely a pass which would change the structure of the code, reordering blocks and changing conditions.

After reading the description of all switches in the detailed gcc manual, the -ftree-* pass family looked promising. This set of transformations works on the SSA form internal representation, a widespread intermediate language representation which has the benefit of being easy to read. They seem to make a huge impact on the generated assembly code, moving code blocks around and making assumptions on code invariants in order to move around, simplify or eliminate conditional checks altogether.

By looking at output of those passes on the related source code, we narrowed down the list of transformations to a couple of interesting passes, one of them being -ftree-vrp, which stands for Value Range Propagation. This pass computes bounds for each name binding and propagates proofs that a value must lie in a given range.

It turned out most of the other passes depended on it for further optimisations. Even though the issue ended up not being a bad assumptions in the range values, checking this pass proved to be worthwhile: enabling -ftree-vrp on sweep_slice function while every thing else was built with -O1 was enough to trigger a crash.

GCC provides very good diagnostics output, and after reading the manual we found the -fdump-tree-* switch to dump the SSA form before and after specific pass. The output is designed to be read by a human and provides meaningful naming, with source code locations, alongside the ranges propagated by the VRP pass. We spent some time studying the output and matched the difference in SSA tree between the crashing and not crashing code.

Examining the bounds and invariants derived by gcc, it was clear that no wrong hypothesis was stated.

https://medium.com/media/a5a311d0b5a4f890f9541f0aed91e73e/href

The only meaningful observable change involves the suppression of rechecking the loop condition in the else branch of the sweep_slice function, after Value Range Propagation proved that the condition was invariant in this branch.

Often, reading the code carefully is the fastest way to find a bug. But after spending hours staring at the major GC code, it was clear enough that this check removal should not cause any semantic changes.

In this process, we identified a suspicious bit of code, where a signed long variable was promoted to unsigned according to C standard rules, which was changing the bounds derived by gcc, assuming it was always positive. But after some thinking we realised it made no difference at assembly level and although wrong, this assumption was not used anywhere.

We were now ready to rule out the possibility of a bug in OCaml runtime. It was still possible that GCC backend had a bug and was miscompiling this particular shape of code. And we were back at the assembly level again. After writing some awk formatting script to cleanup assembly and minimise noise in the diff (by renaming labels, detecting spurious code move, etc), and preventing inlining, we found a minimal assembly patch causing the crash.

There were only cosmetic differences. The test removal was propagated down to assembly and caused gcc to reorganise the layout of each switch case block.

https://medium.com/media/532ab4896b11177a96203fd0c81eaa58/href

Among those minor differences and changes of layout, we noticed a particular change which impacted exactly the reachable block header updated which could have caused header corruption. In the unoptimised version, the updating code looked like this:

https://medium.com/media/38ca30a1decf86233084721c3803b1c4/href

For some reason, the block pointer was spilled to the stack.

Perhaps naively, and because we had earlier emitted the hypothesis of HT impacting cache pressure, we spent a few hours staring at this code and check if we were missing something subtle which could affect the control flow of the whole function and the stack location from which it was reloaded could be corrupted.

Despite our lack of assembly knowledge, after spending several hours reading this tiny change, we got convinced that it made strictly no semantic difference. Reading the x86 manual carefully didn’t give any hint on any subtle behavior which would trigger. Executing any of those two sequence of instruction should give the exact same output.

Mitigating the issue

We were now quite certain it was a CPU bug.

The OCaml developers had reached the same conclusion, and were working on escalating the issue to Intel. After internal discussions we decided to keep this bug as low profile as possible since we were unsure about potential security implications, especially for JIT implementations.

Even if we had no confirmation at this point nor any explanations of the cause of this bug, which was beyond our reach, we could take actions.

The first step was to decide against getting any new Skylake based servers until further announcement. We were left with several Skylake machines but we refrained from deploying any OCaml code on them. OCaml comes with a great package manager, opam, which supports compiler switches. Switches allow to set up a clean and distinct environment with specific packages and compiler configuration.

We patched our internal opam repository to distribute unoptimised runtime to all developers and moved forward, waiting for further announcements.

This situation made us realise that microcode requires constant updates, just like any other software in the stack. We raised awareness on this topic in our devops team, and they took measure to ensure we could roll out updates to prod easily.

Happy end

In late May, devops team noticed a debian package update for intel-microcode containing the following change:

Likely fix nightmare-level Skylake erratum SKL150. Fortunately,
either this erratum is very-low-hitting, or gcc/clang/icc/msvc
won’t usually issue the affected opcode pattern and it ends up
being rare.
SKL150 — Short loops using both the AH/BH/CH/DH registers and
the corresponding wide register *may* result in unpredictable
system behavior. Requires both logical processors of the same
core (i.e. sibling hyperthreads) to be active to trigger, as
well as a “complex set of micro-architectural conditions”

The erratum description immediately rang a bell as it matched the diff in the assembly we had observed. We tested the microcode update and confirmed it fixed the corruption.

Finally, our Skylake CPUs were feeling safe and OCaml compiler was happy.

Ahrefs runs an internet-scale bot that crawls the whole Web 24/7. Our backend system is powered by a custom petabyte-scale distributed key-value storage implemented in OCaml (and some C++ and Rust). We are a small team and strongly believe in better technology leading to better solutions for real-world problems. We worship functional languages and static typing, extensively employ code generation and meta-programming, value code clarity and predictability, and are constantly seeking to automate repetitive tasks and eliminate boilerplate. And we are hiring!

Skylake bug: a detective story was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deploying reproducible unikernels with albatross

hannes — 2022-11-17T12:41:11-00:00

Deploying MirageOS unikernels

More than five years ago, I posted how to deploy MirageOS unikernels. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described how to deploy reproducible unikernels.

Albatross

In recent months we worked hard on the underlying infrastructure: albatross. Albatross is the orchestration system for MirageOS unikernels that use solo5 with hvt or spt tender. It deals with three tasks:

unikernel creation (destroyal, restart) +
capturing console output +
collecting metrics in the host system about unikernels +

An addition to the above is dealing with multiple tenants on the same machine: remote management of your unikernel fleet via TLS, and resource policies.

History

The initial commit of albatross was in May 2017. Back then it replaced the shell scripts and manual scp of unikernel images to the server. Over time it evolved and adapted to new environments. Initially a solo5 unikernel would only know of a single network interface, these days there can be multiple distinguished by name. Initially there was no support for block devices. Only FreeBSD was supported in the early days. Nowadays we built daily packages for Debian, Ubuntu, FreeBSD, and have support for NixOS, and the client side is supported on macOS as well.

ASN.1

The communication format between the albatross daemons and clients was changed multiple times. I'm glad that albatross uses ASN.1 as communication format, which makes extension with optional fields easy, and also allows "choice" (the sum type) to be not tagged (the binary is the same as no choice type), thus adding choice to an existing grammar, and preserving the old in the default (untagged) case is a decent solution.

So, if you care about backward and forward compatibility, as we do, since we may be in control of which albatross servers are deployed on our machine, but not what albatross versions the clients are using -- it may be wise to look into ASN.1. Recent efforts (json with schema, ...) may solve similar issues, but ASN.1 is as well very tiny in size.

What resources does a unikernel need?

A unikernel is just an operating system for a single service, there can't be much it can need.

Name

So, first of all a unikernel has a name, or a handle. This is useful for reporting statistics, but also to specify which console output you're interested in. The name is a string with printable ASCII characters (and dash '-' and dot '.'), with a length up to 64 characters - so yes, you can use an UUID if you like.

Memory

Another resource is the amount of memory assigned to the unikernel. This is specified in megabyte (as solo5 does), with the range being 10 (below not even a hello world wants to start) to 1024.

Arguments

Of course, you can pass via albatross boot parameters to the unikernel. Albatross doesn't impose any restrictions here, but the lower levels may.

CPU

Due to multiple tenants, and side channel attacks, it looked right at the beginning like a good idea to restrict each unikernel to a specific CPU. This way, one tenant may use CPU 5, and another CPU 9 - and they'll not starve each other (best to make sure that these CPUs are in different packages). So, albatross takes a number as the CPU, and executes the solo5 tender within taskset/cpuset.

Fail behaviour

In normal operations, exceptional behaviour may occur. I have to admit that I've seen MirageOS unikernels that suffer from not freeing all the memory they have allocated. To avoid having to get up at 4 AM just to start the unikernel that went out of memory, there's the possibility to restart the unikernel when it exited. You can even specify on which exit codes it should be restarted (the exit code is the only piece of information we have from the outside what caused the exit). This feature was implemented in October 2019, and has been very precious since then. :)

Network

This becomes a bit more complex: a MirageOS unikernel can have network interfaces, and solo5 specifies a so-called manifest with a list of these (name and type, and type is so far always basic). Then, on the actual server there are bridges (virtual switches) configured. Now, these may have the same name, or may need to be mapped. And of course, the unikernel expects a tap interface that is connected to such a bridge, not the bridge itself. Thus, albatross creates tap devices, attaches these to the respective bridges, and takes care about cleaning them up on teardown. The albatross client verifies that for each network interface in the manifest, there is a command-line argument specified (--net service:my_bridge or just --net service if the bridge is named service). The tap interface name is not really of interest to the user, and will not be exposed.

Block devices

On the host system, it's just a file, and passed to the unikernel. There's the need to be able to create one, dump it, and ensure that each file is only used by one unikernel. That's all that is there.

Metrics

Everyone likes graphs, over time, showing how much traffic or CPU or memory or whatever has been used by your service. Some of these statistics are only available in the host system, and it is also crucial for development purposes to compare whether the bytes sent in the unikernel sum up to the same on the host system's tap interface.

The albatross-stats daemon collects metrics from three sources: network interfaces, getrusage (of a child process), VMM debug counters (to count VM exits etc.). Since the recent 1.5.3, albatross-stats now connects at startup to the albatross-daemon and then retrieves the information which unikernels are up and running, and starts periodically collecting data in memory.

Other clients, being it a dump on your console window, a write into an rrd file (good old MRTG times), or a push to influx, can use the stats data to correlate and better analyse what is happening on the grand scale of things. This helped a lot by running several unikernels with different opam package sets to figure out which opam packages leave their hands on memory over time.

As a side note, if you make the unikernel name also available in the unikernel, it can tag its own metrics with the same identifier, and you can correlate high-level events (such as amount of HTTP requests) with low-level things "allocated more memory" or "consumed a lot of CPU".

Console

There's not much to say about the console, just that the albatross-console daemon is running with low privileges, and reading from a FIFO that the unikernel writes to. It never writes anything to disk, but keeps the last 1000 lines in memory, available from a client asking for it.

The daemons

So, the main albatross-daemon runs with superuser privileges to create virtual machines, and opens a unix domain socket where the clients and other daemons are connecting to. The other daemons are executed with normal user privileges, and never write anything to disk.

The albatross-daemon keeps state about the running unikernels, and if it is restarted, the unikernels are started again. Maybe worth to mention that this lead sometimes to headaches (due to data being dumped to disk, and the old format should always be supported), but was also a huge relief to not have to care about creating all the unikernels just because albatross-daemon was killed.

Remote management

There's one more daemon program, either albatross-tls-inetd (to be executed by inetd), or albatross-tls-endpoint. They accept clients via a remote TCP connection, and establish a mutual-authenticated TLS handshake. When done, they forward the command to the respective Unix domain socket, and send back the reply.

The daemon itself has a X.509 certificate to authenticate, but the client is requested to show its certificate chain as well. This by now requires TLS 1.3, so the client certificates are sent over the encrypted channel.

A step back, x X.509 certificate contains a public key and a signature from one level up. When the server knows about the root (or certificate authority (CA)) certificate, and following the chain can verify that the leaf certificate is valid. Additionally, a X.509 certificate is a ASN.1 structure with some fixed fields, but also contains extensions, a key-value store where the keys are object identifiers, and the values are key-dependent data. Also note that this key-value store is cryptographically signed.

Albatross uses the object identifier, assigned to Camelus Dromedarius (MirageOS - 1.3.6.1.4.1.49836.42) to encode the command to be executed. This means that once the TLS handshake is established, the command to be executed is already transferred.

In the leaf certificate, there may be the "create unikernel" command with the unikernel image, it's boot parameters, and other resources. Or a "read the console of my unikernel". In the intermediate certificates (from root to leaf), resource policies are encoded (this path may only have X unikernels running with a total of Y MB memory, and Z MB of block storage, using CPUs A and B, accessing bridges C and D). From the root downwards these policies may only decrease. When a unikernel should be created (or other commands are executed), the policies are verified to hold. If they do not, an error is reported.

Fleet management

Of course it is very fine to create your locally compiled unikernel to your albatross server, go for it. But in terms of "what is actually running here?" and "does this unikernel need to be updated because some opam package had a security issues?", this is not optimal.

Since we provide daily reproducible builds with the current HEAD of the main opam-repository, and these unikernels have no configuration embedded (but take everything as boot parameters), we just deploy them. They come with the information what opam packages contributed to the binary, which environment variables were set, and which system packages were installed with which versions.

The whole result of reproducible builds for us means: we have a hash of a unikernel image that we can lookup in our build infrastructure, and take a look whether there is a newer image for the same job. And if there is, we provide a diff between the packages contributed to the currently running unikernel and the new image. That is what the albatross-client update command is all about.

Of course, your mileage may vary and you want automated deployments where each git commit triggers recompilation and redeployment. The downside would be that sometimes only dependencies are updated and you've to cope with that.

At the moment, there is a client connecting directly to the unix domain sockets, albatross-client-local, and one connecting to the TLS endpoint, albatross-client-bistro. The latter applies compression to the unikernel image.

Installation

For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to /etc/apt/sources.list:

$ wget -q -O /etc/apt/trusted.gpg.d/apt.robur.coop.gpg https://apt.robur.coop/gpg.pub
+$ echo "deb https://apt.robur.coop ubuntu-20.04 main" >> /etc/apt/sources.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
+$ apt update
+$ apt install solo5 albatross
+

On FreeBSD:

$ fetch -o /usr/local/etc/pkg/robur.pub https://pkg.robur.coop/repo.pub # download RSA public key
+$ echo 'robur: {
+  url: "https://pkg.robur.coop/${ABI}",
+  mirror_type: "srv",
+  signature_type: "pubkey",
+  pubkey: "/usr/local/etc/pkg/robur.pub",
+  enabled: yes
+}' > /usr/local/etc/pkg/repos/robur.conf # Check https://pkg.robur.coop which ABI are available
+$ pkg update
+$ pkg install solo5 albatross
+

For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (opam install solo5 albatross). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.

Conclusion

After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.

Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on donations for doing our work - everyone can contribute.

Mirroring the opam repository and all tarballs

hannes — 2022-09-29T13:04:14-00:00

We at robur developed opam-mirror in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).

What is opam and why should I care?

Opam is the OCaml package manager (also used by other projects such as coq). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as ocaml/opam-repository, where authors of OCaml software can contribute (as pull request) their latest releases.

When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state "my package will not work with cmdliner in version 1.2.0").

So, the opam-repository holds the metadata of lots of OCaml packages (around 4000 at the moment this article was written) with lots of versions (in total 25000) that have been released. It is used by the opam client to figure out which packages to install or upgrade (using a solver that takes the version bounds into consideration).

Of course, opam can use other repositories (overlays) or forks thereof. So nothing stops you from using any other opam repository. The url to the source code of each package may be a tarball, or a git repository or other version control systems.

The vast majority of opam packages released to the opam-repository include a link to the source tarball and a cryptographic hash of the tarball. This is crucial for security (under the assumption the opam-repository has been downloaded from a trustworthy source - check back later this year for updates on conex). At the moment, there are some weak spots in respect to security: md5 is still allowed, and the hash and the tarball are downloaded from the same server: anyone who is in control of that server can inject arbitrary malicious data. As outlined above, we're working on infrastructure which fixes the latter issue.

How does the opam client work?

Opam, after initialisation, downloads the index.tar.gz from https://opam.ocaml.org/index.tar.gz, and uses this as the local opam universe. An opam install cmdliner will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e.

How does the opam repository work?

According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses opam admin index periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was a reported downtime.

Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS hack retreats in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).

Re-developing "opam admin create" as MirageOS unikernel

The need for a local opam cache at our reproducible build infrastructure and the retreats, we decided to develop opam-mirror as a MirageOS unikernel. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the opam admin index is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.

Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.

In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with crunch, or using a remote git repository), but recent development, e.g. of chamelon sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.

Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.

Code development and improvements

Initially, our plan was to use ocaml-git for pulling the repository, chamelon for persistent storage, and httpaf as web server. With ocaml-tar recent support of gzip we should be all set, and done within a few days.

There is already a gap in the above plan: which http client to use - in the best case something similar to our http-lwt-client - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using happy-eyeballs to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.

On the path (over the last month), we fixed file descriptor leaks (memory leaks) in paf -- which is used as a runtime for httpaf and h2.

Then we ran into some trouble with chamelon (out of memory, some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.

We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed git-kv which can dump and restore the git state.

Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.

Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as fat may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed mirage-block-partition to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed oneffs which stores a single value on the block device.

We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains recent enhancements of the mirage-kv API, there is a get_partial, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).

What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy tlstunnel) similar to how unipi does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).

Conclusion

To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide reproducible binaries for solo5-hvt. You can use our opam mirror with opam repository set-url default https://opam.robur.coop (revert to the other with opam repository set-url default https://opam.ocaml.org) or use it as a backup with opam repository add robur --rank 2 https://opam.robur.coop.

Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on donations for doing our work - everyone can contribute.

All your metrics belong to influx

hannes — 2022-03-08T11:26:31-00:00

Introduction to monitoring

At robur we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our reproducible builds infrastructure. Another part is, once deployed we want to observe what is going on.

I first got into touch with monitoring - collecting and graphing metrics - with MRTG and munin - and the simple network management protocol SNMP. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.

The rise of monitoring where graphs are done dynamically (such as Grafana) and can be programmed (with a query language) by the operator are very neat, it allows to put metrics in relation after they have been recorded - thus if there's a thesis why something went berserk, you can graph the collected data from the past and prove or disprove the thesis.

Monitoring a MirageOS unikernel

From the operational perspective, taking security into account - either the data should be authenticated and integrity-protected, or being transmitted on a private network. We chose the latter, there's a private network interface only for monitoring. Access to that network is only granted to the unikernels and metrics collector.

For MirageOS unikernels, we use the metrics library - which design shares the idea of logs that only if there's a reporter registered, work is performed. We use the Influx line protocol via TCP to report via Telegraf to InfluxDB. But due to the design of metrics, other reporters can be developed and used -- prometheus, SNMP, your-other-favourite are all possible.

Apart from monitoring metrics, we use the same network interface for logging via syslog. Since the logs library separates the log message generation (in the OCaml libraries) from the reporting, we developed logs-syslog, which registers a log reporter sending each log message to a syslog sink.

We developed a small library for metrics reporting of a MirageOS unikernel into the monitoring-experiments package - which also allows to dynamically adjust log level and disable or enable metrics sources.

Required components

Install from your operating system the packages providing telegraf, influxdb, and grafana.

Setup telegraf to contain a socket listener:

[[inputs.socket_listener]]
+  service_address = "tcp://192.168.42.14:8094"
+  keep_alive_period = "5m"
+  data_format = "influx"
+

Use a unikernel that reports to Influx (below the heading "Unikernels (with metrics reported to Influx)" on builds.robur.coop) and provide --monitor=192.168.42.14 as boot parameter. Conventionally, these unikernels expect a second network interface (on the "management" bridge) where telegraf (and a syslog sink) are running. You'll need to pass --net=management and --arg='--management-ipv4=192.168.42.x/24' to albatross-client-local.

Albatross provides a albatross-influx daemon that reports information from the host system about the unikernels to influx. Start it with --influx=192.168.42.14.

Adding monitoring to your unikernel

If you want to extend your own unikernel with metrics, follow along these lines.

An example is the dns-primary-git unikernel, where on the branch future we have a single commit ahead of main that adds monitoring. The difference is in the unikernel configuration and the main entry point. See the binary builts in contrast to the non-monitoring builts.

In config, three new command line arguments are added: --monitor=IP, --monitor-adjust=PORT --syslog=IP and --name=STRING. In addition, the package monitoring-experiments is required. And a second network interface management_stack using the prefix management is required and passed to the unikernel. Since the syslog reporter requires a console (to report when logging fails), also a console is passed to the unikernel. Each reported metrics includes a tag vm=<name> that can be used to distinguish several unikernels reporting to the same InfluxDB.

Command line arguments:

   let doc = Key.Arg.info ~doc:"The fingerprint of the TLS certificate." [ "tls-cert-fingerprint" ] in
+   Key.(create "tls_cert_fingerprint" Arg.(opt (some string) None doc))
+ 
++let monitor =
++  let doc = Key.Arg.info ~doc:"monitor host IP" ["monitor"] in
++  Key.(create "monitor" Arg.(opt (some ip_address) None doc))
++
++let monitor_adjust =
++  let doc = Key.Arg.info ~doc:"adjust monitoring (log level, ..)" ["monitor-adjust"] in
++  Key.(create "monitor_adjust" Arg.(opt (some int) None doc))
++
++let syslog =
++  let doc = Key.Arg.info ~doc:"syslog host IP" ["syslog"] in
++  Key.(create "syslog" Arg.(opt (some ip_address) None doc))
++
++let name =
++  let doc = Key.Arg.info ~doc:"Name of the unikernel" ["name"] in
++  Key.(create "name" Arg.(opt string "ns.nqsb.io" doc))
++
+ let mimic_impl random stackv4v6 mclock pclock time =
+   let tcpv4v6 = tcpv4v6_of_stackv4v6 $ stackv4v6 in
+   let mhappy_eyeballs = mimic_happy_eyeballs $ random $ time $ mclock $ pclock $ stackv4v6 in
+

Requiring monitoring-experiments, registering command line arguments:

     package ~min:"3.7.0" ~max:"3.8.0" "git-mirage";
+     package ~min:"3.7.0" "git-paf";
+     package ~min:"0.0.8" ~sublibs:["mirage"] "paf";
++    package "monitoring-experiments";
++    package ~sublibs:["mirage"] ~min:"0.3.0" "logs-syslog";
+   ] in
+   foreign
+-    ~keys:[Key.abstract remote_k ; Key.abstract axfr]
++    ~keys:[
++      Key.abstract remote_k ; Key.abstract axfr ;
++      Key.abstract name ; Key.abstract monitor ; Key.abstract monitor_adjust ; Key.abstract syslog
++    ]
+     ~packages
+

Added console and a second network stack to foreign:

     "Unikernel.Main"
+-    (random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> job)
++    (console @-> random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> stackv4v6 @-> job)
++
+

Passing a console implementation (default_console) and a second network stack (with management prefix) to register:

+let management_stack = generic_stackv4v6 ~group:"management" (netif ~group:"management" "management")
+ 
+ let () =
+   register "primary-git"
+-    [dns_handler $ default_random $ default_posix_clock $ default_monotonic_clock $
+-     default_time $ net $ mimic_impl]
++    [dns_handler $ default_console $ default_random $ default_posix_clock $ default_monotonic_clock $
++     default_time $ net $ mimic_impl $ management_stack]
+

Now, in the unikernel module the functor changes (console and second network stack added):

@@ -4,17 +4,48 @@
+ 
+ open Lwt.Infix
+ 
+-module Main (R : Mirage_random.S) (P : Mirage_clock.PCLOCK) (M : Mirage_clock.MCLOCK) (T : Mirage_time.S) (S : Mirage_stack.V4V6) (_ : sig e
+nd) = struct
++module Main (C : Mirage_console.S) (R : Mirage_random.S) (P : Mirage_clock.PCLOCK) (M : Mirage_clock.MCLOCK) (T : Mirage_time.S) (S : Mirage
+_stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
+ 
+   module Store = Irmin_mirage_git.Mem.KV(Irmin.Contents.String)
+   module Sync = Irmin.Sync(Store)
+

And in the start function, the command line arguments are processed and used to setup syslog and metrics monitoring to the specified addresses. Also, a TCP listener is waiting for monitoring and logging adjustments if --monitor-adjust was provided:

   module D = Dns_server_mirage.Make(P)(M)(T)(S)
++  module Monitoring = Monitoring_experiments.Make(T)(Management)
++  module Syslog = Logs_syslog_mirage.Udp(C)(P)(Management)
+ 
+-  let start _rng _pclock _mclock _time s ctx =
++  let start c _rng _pclock _mclock _time s ctx management =
++    let hostname = Key_gen.name () in
++    (match Key_gen.syslog () with
++     | None -> Logs.warn (fun m -> m "no syslog specified, dumping on stdout")
++     | Some ip -> Logs.set_reporter (Syslog.create c management ip ~hostname ()));
++    (match Key_gen.monitor () with
++     | None -> Logs.warn (fun m -> m "no monitor specified, not outputting statistics")
++     | Some ip -> Monitoring.create ~hostname ?listen_port:(Key_gen.monitor_adjust ()) ip management);
+     connect_store ctx >>= fun (store, upstream) ->
+     load_git None store upstream >>= function
+     | Error (`Msg msg) ->
+

Once you compiled the unikernel (or downloaded a binary with monitoring), and start that unikernel by passing --net:service=tap0 and --net:management=tap10 (or whichever your tap interfaces are), and as unikernel arguments --ipv4=<my-ip-address> and --management-ipv4=192.168.42.2/24 for IPv4 configuration, --monitor=192.168.42.14, --syslog=192.168.42.10, --name=my.unikernel, --monitor-adjust=12345.

With this, your unikernel will report metrics using the influx protocol to 192.168.42.14 on port 8094 (every 10 seconds), and syslog messages via UDP to 192.168.0.10 (port 514). You should see your InfluxDB getting filled and syslog server receiving messages.

When you configure Grafana to use InfluxDB, you'll be able to see the data in the data sources.

Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.

Deploying binary MirageOS unikernels

hannes — 2021-06-30T13:13:37-00:00

Introduction

MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish our goal to "get MirageOS into production", we need to lower the barrier. This means for us to release binary unikernels. As described earlier, we received a grant for "Deploying MirageOS" from NGI Pointer to work on the required infrastructure. This is joint work with Reynir.

We provide at builds.robur.coop binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be reproducible. This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.

This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.

Deploying MirageOS unikernels from binary

To execute a MirageOS unikernel, apart from a hypervisor (Xen/KVM/Muen), a tender (responsible for allocating host system resources and passing these to the unikernel) is needed. Using virtio, this is conventionally done with qemu on Linux, but its code size (and attack surface) is huge. For MirageOS, we develop Solo5, a minimal tender. It supports hvt - hardware virtualization (Linux KVM, FreeBSD BHyve, OpenBSD VMM), spt - sandboxed process (a tight seccomp ruleset (only a handful of system calls allowed, no hardware virtualization needed), Linux only). Apart from that, muen (a hypervisor developed in Ada), virtio (for some cloud deployments), and xen (PVHv2 or Qubes 4.0) - read more. We deploy our unikernels as hvt with FreeBSD BHyve as hypervisor.

On builds.robur.coop, next to the unikernel images, solo5-hvt packages are provided - download the binary and install it. A NixOS package is already available - please note that soon packaging will be much easier (and we will work on packages merged into distributions).

When the tender is installed, download a unikernel image (e.g. the traceroute described in an earlier post), and execute it:

$ solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1
+

If you plan to orchestrate MirageOS unikernels, you may be interested in albatross - we provide binary packages as well for albatross. An upcoming post will go into further details of how to setup albatross.

MirageOS configuration

A MirageOS unikernel has a specific purpose - composed of OCaml libraries - selected at compile time, which allows to only embed the required pieces. This reduces the attack surface drastically. At the same time, to be widely useful to multiple organisations, no configuration data must be embedded into the unikernel.

Early MirageOS unikernels such as mirage-www embed content (blog posts, ..) and TLS certificates and private keys in the binary (using crunch). The Qubes firewall (read the blog post by Thomas for more information) used to include the firewall rules until v0.6 in the binary, since v0.7 the rules are read dynamically from QubesDB. This is big usability improvement.

We have several possibilities to provide configuration information in MirageOS, on the one hand via boot parameters (can be pre-filled at development time, and further refined at configuration time, but those passed at boot time take precedence). Boot parameters have a length limitation.

Another option is to use a block device - where the TLS reverse proxy stores the configuration, modifiable via a TCP control socket (authentication using a shared hmac secret).

Several other unikernels, such as this website and our CalDAV server, store the content in a remote git repository. The git URI and credentials (private key seed, host key fingerprint) are passed via boot parameter.

Finally, another option that we take advantage of is to introduce a post-link step that rewrites the binary to embed configuration. The tool caravan developed by Romain that does this rewrite is used by our openvpn router (binary).

In the future, some configuration information - such as monitoring system, syslog sink, IP addresses - may be done via DHCP on one of the private network interfaces - this would mean that the DHCP server has some global configuration option, and the unikernels no longer require that many boot parameters. Another option we want to investigate is where the tender shares a file as read-only memory-mapped region from the host system to the guest system - but this is tricky considering all targets above (especially virtio and muen).

Behind the scenes: reproducible builds

To provide a high level of assurance and trust, if you distribute binaries in 2021, you should have a recipe how they can be reproduced in a bit-by-bit identical way. This way, different organisations can run builders and rebuilders, and a user can decide to only use a binary if it has been reproduced by multiple organisations in different jurisdictions using different physical machines - to avoid malware being embedded in the binary.

For a reproduction to be successful, you need to collect the checksums of all sources that contributed to the built, together with other things (host system packages, environment variables, etc.). Of course, you can record the entire OS and sources as a tarball (or file system snapshot) and distribute that - but this may be suboptimal in terms of bandwidth requirements.

With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the opam switch export includes extra-files (patches) and records the VCS version. Based on this functionality, orb, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are PATH (sanitized with an allow list, /bin, /sbin, with /usr, /usr/local, and /opt prefixes), and HOME. To enable Debian's apt to install packages, DEBIAN_FRONTEND is set to noninteractive. The SWITCH_PATH is recorded to allow orb to use the same path during a rebuild. The SOURCE_DATE_EPOCH is set to enable tools that record a timestamp to use a static one. The OS* variables are only used for recording the host OS and version.

The goal of reproducible builds can certainly be achieved in several ways, including to store all sources and used executables in a huge tarball (or docker container), which is preserved for rebuilders. The question of minimal trusted computing base and how such a container could be rebuild from sources in reproducible way are open.

The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.

For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. hvt), and we occasionally development versions of opam packages, we use the unikernel-repo as overlay.

If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.

Each unikernel (and non-unikernel) job (e.g. dns-primary outputs some artifacts:

the binary image (in bin/, unikernel image, OS package) +
the build-environment containing the environment variables used for this build +
the system-packages containing all packages installed on the host system +
the opam-switch that contains all opam packages, including git commit or tarball with checksum, and potentially extra patches, used for this build +
a job script and console output +

To reproduce such a built, you need to get the same operating system (OS, OS_FAMILY, OS_DISTRIBUTION, OS_VERSION in build-environment), the same set of system packages, and then you can orb rebuild which sets the environment variables and installs the opam packages from the opam-switch.

You can browse the different builds, and if there are checksum changes, you can browse to a diff between the opam switches to reason whether the checksum change was intentional (e.g. here the checksum of the unikernel changed when the x509 library was updated).

The opam reproducible build infrastructure is driven by:

orb conducting reproducible builds (packages) +
builder scheduling builds in contained environments (packages) +
builder-web storing builds in a database and providing a HTTP interface (packages) +

These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.

Conclusion

Thanks to NGI funding we now have reproducible MirageOS binary builds available at builds.robur.coop. The underlying infrastructure is reproducible, available for multiple platforms (Ubuntu using docker, FreeBSD using jails), and can be easily bootstrapped from source (once you have OCaml and opam working, getting builder and orb should be easy). All components are open source software, mostly with permissive licenses.

We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.

We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.

Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.

Cryptography updates in OCaml and MirageOS

hannes — 2021-04-23T13:33:06-00:00

Introduction

Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified fiat-crypto stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.

Elliptic curve cryptography

Since May 2020, our OCaml-TLS stack supports TLS 1.3 (since tls version 0.12.0 on opam).

TLS 1.3 requires elliptic curve cryptography - which was not available in mirage-crypto (the maintained fork of nocrypto).

There are two major uses of elliptic curves: key exchange (ECDH) for establishing a shared secret over an insecure channel, and digital signature (ECDSA) for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)

Elliptic curve cryptoraphy is vulnerable to various timing attacks - have a read of the overview article on ECDSA. When implementing elliptic curve cryptography, it is best to avoid these known attacks. Gladly, there are some projects which address these issues by construction.

In addition, to use the code in MirageOS, it should be boring C code: no heap allocations, only using a very small amount of C library functions -- the code needs to be compiled in an environment with nolibc.

Two projects started in semantics, to solve the issue from the grounds up: fiat-crypto and hacl-star: their approach is to use a proof system (Coq or F* to verify that the code executes in constant time, not depending on data input. Both projects provide as output of their proof systems C code.

For our initial TLS 1.3 stack, Clément, Nathan and Etienne developed fiat-p256 and hacl_x5519. Both were one-shot interfaces for a narrow use case (ECDH for NIST P-256 and X25519), worked well for their purpose, and allowed to gather some experience from the development side.

Changed requirements

Revisiting our cryptography stack with the elliptic curve perspective had several reasons, on the one side the customer project NetHSM asked for feasibility of ECDSA/EdDSA for various elliptic curves, on the other side DNSSec uses elliptic curve cryptography (ECDSA), and also wireguard relies on elliptic curve cryptography. The number of X.509 certificates using elliptic curves is increasing, and we don't want to leave our TLS stack in a state where it can barely talk to a growing number of services on the Internet.

Looking at hacl-star, their support is limited to P-256 and Curve25519, any new curve requires writing F*. Another issue with hacl-star is C code quality: their C code does neither compile with older C compilers (found on Oracle Linux 7 / CentOS 7), nor when enabling all warnings (> 150 are generated). We consider the C compiler as useful resource to figure out undefined behaviour (and other problems), and when shipping C code we ensure that it compiles with -Wall -Wextra -Wpedantic --std=c99 -Werror. The hacl project ships a bunch of header files and helper functions to work on all platforms, which is a clunky ifdef desert. The hacl approach is to generate a whole algorithm solution: from arithmetic primitives, group operations, up to cryptographic protocol - everything included.

In contrast, fiat-crypto is a Coq development, which as part of compilation (proof verification) generates executables (via OCaml code extraction from Coq). These executables are used to generate modular arithmetic (as C code) given a curve description. The generated C code is highly portable, independent of platform (word size is taken as input) - it only requires a <stdint.h>, and compiles with all warnings enabled (once a minor PR got merged). Supporting a new curve is simple: generate the arithmetic code using fiat-crypto with the new curve description. The downside is that group operations and protocol needs to implemented elsewhere (and is not part of the proven code) - gladly this is pretty straightforward to do, especially in high-level languages.

Working with fiat-crypto

As mentioned, our initial fiat-p256 binding provided ECDH for the NIST P-256 curve. Also, BoringSSL uses fiat-crypto for ECDH, and developed the code for group operations and cryptographic protocol on top of it.

The work needed was (a) ECDSA support and (b) supporting more curves (let's focus on NIST curves). For ECDSA, the algorithm requires modular arithmetics in the field of the group order (in addition to the prime). We generate these primitives with fiat-crypto (named npYYY_AA) - that required a small fix in decoding hex. Fiat-crypto also provides inversion since late October 2020, paper - which allowed to reduce our code base taken from BoringSSL. The ECDSA protocol was easy to implement in OCaml using the generated arithmetics.

Addressing the issue of more curves was also easy to achieve, the C code (group operations) are macros that are instantiated for each curve - the OCaml code are functors that are applied with each curve description.

Thanks to the test vectors (as structured data) from wycheproof (and again thanks to Etienne, Nathan, and Clément for their OCaml code decodin them), I feel confident that our elliptic curve code works as desired.

What was left is X25519 and Ed25519 - dropping the hacl dependency entirely felt appealing (less C code to maintain from fewer projects). This turned out to require more C code, which we took from BoringSSL. It may be desirable to reduce the imported C code, or to wait until a project on top of fiat-crypto which provides proven cryptographic protocols is in a usable state.

To avoid performance degradation, I distilled some X25519 benchmarks, turns out the fiat-crypto and hacl performance is very similar.

Achievements

The new opam package mirage-crypto-ec is released, which includes the C code generated by fiat-crypto (including inversion), point operations from BoringSSL, and some OCaml code for invoking these functions and doing bounds checks, and whether points are on the curve. The OCaml code are some functors that take the curve description (consisting of parameters, C function names, byte length of value) and provide Diffie-Hellman (Dh) and digital signature algorithm (Dsa) modules. The nonce for ECDSA is computed deterministically, as suggested by RFC 6979, to avoid private key leakage.

The code has been developed in NIST curves, removing blinding (since we use operations that are verified to be constant-time), added missing length checks (reported by Greg), curve25519, a fix for signatures that do not span the entire byte size (discovered while adapting X.509), fix X25519 when the input has offset <> 0. It works on x86 and arm, both 32 and 64 bit (checked by CI). The development was partially sponsored by Nitrokey.

What is left to do, apart from further security reviews, is performance improvements, Ed448/X448 support, and investigating deterministic k for P521. Pull requests are welcome.

When you use the code, and encounter any issues, please report them.

Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed

With the sign and verify primitives, the next step is to interoperate with other tools that generate and use these public and private keys. This consists of serialisation to and deserialisation from common data formats (ASN.1 DER and PEM encoding), and support for handling X.509 certificates with elliptic curve keys. Since X.509 0.12.0, it supports EC private and public keys, including certificate validation and issuance.

Releasing X.509 also included to go through the issue tracker and attempt to solve the existing issues. This time, the "country name is encoded as UTF8String, while RFC demands PrintableString" filed more than 5 years ago by Reynir, re-reported by Petter in early 2017, and again by Vadim in late 2020, was fixed by Vadim.

Another long-standing pull request was support for PKCS 12, the archive format for certificate and private key bundles. This has been developed and merged. PKCS 12 is a widely used and old format (e.g. when importing / exporting cryptographic material in your browser, used by OpenVPN, ...). Its specification uses RC2 and 3DES (see this nice article), which are the default algorithms used by openssl pkcs12.

One more layer up - TLS

In TLS we are finally able to use ECDSA (and EdDSA) certificates and private keys, this resulted in slightly more complex configuration - the constraints between supported groups, signature algorithms, ciphersuite, and certificates are intricate:

The ciphersuite (in TLS before 1.3) specifies which key exchange mechanism to use, but also which signature algorithm to use (RSA/ECDSA). The supported groups client hello extension specifies which elliptic curves are supported by the client. The signature algorithm hello extension (TLS 1.2 and above) specifies the signature algorithm. In the end, at load time the TLS configuration is validated and groups, ciphersuites, and signature algorithms are condensed depending on configured server certificates. At session initiation time, once the client reports what it supports, these parameters are further cut down to eventually find some suitable cryptographic parameters for this session.

From the user perspective, earlier the certificate bundle and private key was a pair of X509.Certificate.t list and Mirage_crypto_pk.Rsa.priv, now the second part is a X509.Private_key.t - all provided constructors have been updates (notably X509_lwt.private_of_pems and Tls_mirage.X509.certificate).

Finally, conduit and mirage

Thanks to Romain, conduit* 4.0.0 was released which supports the modified API of X.509 and TLS. Romain also developed patches and released mirage 3.10.3 which supports the above mentioned work.

Conclusion

Elliptic curve cryptography is now available in OCaml using verified cryptographic primitives from the fiat-crypto project - opam install mirage-crypto-ec. X.509 since 0.12.0 and TLS since 0.13.0 and MirageOS since 3.10.3 support this new development which gives rise to smaller EC keys. Our old bindings, fiat-p256 and hacl_x25519 have been archived and will no longer be maintained.

Thanks to everyone involved on this journey: reporting issues, sponsoring parts of the work, helping with integration, developing initial prototypes, and keep motivating me to continue this until the release is done.

In the future, it may be possible to remove zarith and gmp from the dependency chain, and provide EC-only TLS servers and clients for MirageOS. The benefit will be much less C code (libgmp-freestanding.a is 1.5MB in size) in our trusted code base.

Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.

Footer

If you want to support our work on MirageOS unikernels, please donate to robur. I'm interested in feedback, either via twitter, hannesm@mastodon.social or via eMail.

The road ahead for MirageOS in 2021

hannes — 2021-01-25T12:45:54-00:00

Introduction

2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations, and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)

I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.

For 2021, we (at robur) got a grant from the EU (via NGI pointer) for "Deploying MirageOS" (more details below), and another grant from OCaml software foundation for securing the opam supply chain (using conex). Some long-awaited releases for MirageOS libraries, namely a ssh implementation and a rewrite of our git implementation have already been published.

With my MirageOS view, 2020 was a pretty successful year, where we managed to add more features, fixed lots of bugs, and paved the road ahead. I want to thank OCamlLabs for funding work on MirageOS maintenance.

Recap 2020

Here is a very subjective random collection of accomplishments in 2020, where I was involved with some degree.

NetHSM

NetHSM is a hardware security module in software. It is a product that uses MirageOS for security, and is based on the muen separation kernel. We at robur were heavily involved in this product. It already has been security audited by an external team. You can pre-order it from Nitrokey.

TLS 1.3

Dating back to 2016, at the TRON (TLS 1.3 Ready or NOt), we developed a first draft of a 1.3 implementation of OCaml-TLS. Finally in May 2020 we got our act together, including ECC (ECDH P256 from fiat-crypto, X25519 from hacl) and testing with tlsfuzzer, and release tls 0.12.0 with TLS 1.3 support. Later we added ECC ciphersuites to TLS version 1.2, implemented ChaCha20/Poly1305, and fixed an interoperability issue with Go's implementation.

Mirage-crypto provides the underlying cryptographic primitives, initially released in March 2020 as a fork of nocrypto -- huge thanks to pqwy for his great work. Mirage-crypto detects CPU features at runtime (thanks to Julow) (bugfix for bswap), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports compilation on Windows (thanks to avsm), async entropy harvesting (thanks to seliopou), 32 bit support, chacha20/poly1305 (thanks to abeaumont), cross-compilation (thanks to EduardoRFS) and various bug fixes, even memory leak (thanks to talex5 for reporting several of these issues), and RSA interoperability (thanks to psafont for investigation and mattjbray for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.

Qubes Firewall

The MirageOS based Qubes firewall is the most widely used MirageOS unikernel. And it got major updates: in May Steffi announced her and Mindy's work on improving it for Qubes 4.0 - including dynamic firewall rules via QubesDB. Thanks to prototypefund for sponsoring.

In October 2020, we released Mirage 3.9 with PVH virtualization mode (thanks to mato). There's still a memory leak to be investigated and fixed.

IPv6

In December, with Mirage 3.10 we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to nojb for the initial code and MagnusS.

Turns out this blog, but also robur services, are now available via IPv6 :)

Albatross

Also in December, I pushed an initial release of albatross, a unikernel orchestration system with remote access. Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.

Thanks to reynir for statistics support on Linux and improvements of the systemd service scripts. Also thanks to cfcs for the initial Linux port.

CA certs

For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to emillon for initial code, there are now ca-certs and ca-certs-nss opam packages (see release announcement) which fills this gap.

Unikernels

I developed several useful unikernels in 2020, and also pushed a unikernel gallery to the Mirage website:

Traceroute in MirageOS

I already wrote about traceroute which traces the routing to a given remote host.

Unipi - static website hosting

Unipi is a static site webserver which retrieves the content from a remote git repository. Let's encrypt certificate provisioning and dynamic updates via a webhook to be executed for every push.

TLSTunnel - TLS demultiplexing

The physical machine this blog and other robur infrastructure runs on has been relocated from Sweden to Germany mid-December. Thanks to UPS! Fewer IPv4 addresses are available in the new data center, which motivated me to develop tlstunnel.

The new behaviour is as follows (see the monitoring branch):

listener on TCP port 80 which replies with a permanent redirect to https +
listener on TCP port 443 which forwards to a backend host if the requested server name is configured +
its configuration is stored on a block device, and can be dynamically changed (with a custom protocol authenticated with a HMAC) +
it is setup to hold a wildcard TLS certificate and in DNS a wildcard entry is pointing to it +
setting up a new service is very straightforward: only the new name needs to be registered with tlstunnel together with the TCP backend, and everything will just work +

2021

The year started with a release of awa, a SSH implementation in OCaml (thanks to haesbaert for initial code). This was followed by a git 3.0 release (thanks to dinosaure).

Deploying MirageOS - NGI Pointer

For 2021 we at robur received funding from the EU (via NGI pointer) for "Deploying MirageOS", which boils down into three parts:

reproducible binary releases of MirageOS unikernels, +
monitoring (and other devops features: profiling) and integration into existing infrastructure, +
and further documentation and advertisement. +

Of course this will all be available open source. Please get in touch via eMail (team aT robur dot coop) if you're eager to integrate MirageOS unikernels into your infrastructure.

We discovered at an initial meeting with an infrastructure provider that a DNS resolver is of interest - even more now that dnsmasq suffered from dnspooq. We are already working on an implementation of DNSSec.

MirageOS unikernels are binary reproducible, and infrastructure tools are available. We are working hard on a web interface (and REST API - think of it as "Docker Hub for MirageOS unikernels"), and more tooling to verify reproducibility.

Conex - securing the supply chain

Another funding from the OCSF is to continue development and deploy conex - to bring trust into opam-repository. This is a great combination with the reproducible build efforts, and will bring much more trust into retrieving OCaml packages and using MirageOS unikernels.

MirageOS 4.0

Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is close, a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.

Footer

If you want to support our work on MirageOS unikernels, please donate to robur. I'm interested in feedback, either via twitter, hannesm@mastodon.social or via eMail.

Traceroute

hannes — 2020-06-24T10:38:10-00:00

Traceroute

Is a diagnostic utility which displays the route and measures transit delays of +packets across an Internet protocol (IP) network.

$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
+            |      ___|
+  __|  _ \  |  _ \ __ \
+\__ \ (   | | (   |  ) |
+____/\___/ _|\___/____/
+Solo5: Bindings version v0.6.5
+Solo5: Memory map: 512 MB addressable:
+Solo5:   reserved @ (0x0 - 0xfffff)
+Solo5:       text @ (0x100000 - 0x212fff)
+Solo5:     rodata @ (0x213000 - 0x24bfff)
+Solo5:       data @ (0x24c000 - 0x317fff)
+Solo5:       heap >= 0x318000 < stack < 0x20000000
+2020-06-22 15:41:25 -00:00: INF [netif] Plugging into service with mac 76:9b:36:e0:e5:74 mtu 1500
+2020-06-22 15:41:25 -00:00: INF [ethernet] Connected Ethernet interface 76:9b:36:e0:e5:74
+2020-06-22 15:41:25 -00:00: INF [ARP] Sending gratuitous ARP for 10.0.42.2 (76:9b:36:e0:e5:74)
+2020-06-22 15:41:25 -00:00: INF [udp] UDP interface connected on 10.0.42.2
+2020-06-22 15:41:25 -00:00: INF [application]  1  10.0.42.1  351us
+2020-06-22 15:41:25 -00:00: INF [application]  2  192.168.42.1  1.417ms
+2020-06-22 15:41:25 -00:00: INF [application]  3  192.168.178.1  1.921ms
+2020-06-22 15:41:25 -00:00: INF [application]  4  88.72.96.1  16.716ms
+2020-06-22 15:41:26 -00:00: INF [application]  5  *
+2020-06-22 15:41:27 -00:00: INF [application]  6  92.79.215.112  16.794ms
+2020-06-22 15:41:27 -00:00: INF [application]  7  145.254.2.215  21.305ms
+2020-06-22 15:41:27 -00:00: INF [application]  8  145.254.2.217  22.05ms
+2020-06-22 15:41:27 -00:00: INF [application]  9  195.89.99.1  21.088ms
+2020-06-22 15:41:27 -00:00: INF [application] 10  62.115.9.133  20.105ms
+2020-06-22 15:41:27 -00:00: INF [application] 11  213.155.135.82  30.861ms
+2020-06-22 15:41:27 -00:00: INF [application] 12  80.91.246.200  30.716ms
+2020-06-22 15:41:27 -00:00: INF [application] 13  80.91.253.163  28.315ms
+2020-06-22 15:41:27 -00:00: INF [application] 14  62.115.145.27  30.436ms
+2020-06-22 15:41:27 -00:00: INF [application] 15  80.67.4.239  42.826ms
+2020-06-22 15:41:27 -00:00: INF [application] 16  80.67.10.147  47.213ms
+2020-06-22 15:41:27 -00:00: INF [application] 17  198.167.222.207  48.598ms
+Solo5: solo5_exit(0) called
+

This means with a traceroute utility you can investigate which route is taken +to a destination host, and what the round trip time(s) on the path are. The +sample output above is taken from a virtual machine on my laptop to the remote +host 198.167.222.207. You can see there are 17 hops between us, with the first +being my laptop with a tiny round trip time of 351us, the second and third are +using private IP addresses, and are my home network. The round trip time of the +fourth hop is much higher, this is the first hop on the other side of my DSL +modem. You can see various hops on the public Internet: the packets pass from +my Internet provider's backbone across some exchange points to the destination +Internet provider somewhere in Sweden.

The implementation of traceroute relies mainly on the time-to-live (ttl) field +(in IPv6 lingua it is "hop limit") of IP packets, which is meant to avoid route +cycles that would infinitely forward IP packets in circles. Every router, when +forwarding an IP packet, first checks that the ttl field is greater than zero, +and then forwards the IP packet where the ttl is decreased by one. If the ttl +field is zero, instead of forwarding, an ICMP time exceeded packet is sent back +to the source.

Traceroute works by exploiting this mechanism: a series of IP packets with +increasing ttls is sent to the destination. Since upfront the length of the +path is unknown, it is a reactive system: first send an IP packet with a ttl of +one, if a ICMP time exceeded packet is returned, send an IP packet with a ttl of +two, etc. -- until an ICMP packet of type destination unreachable is received. +Since some hosts do not reply with a time exceeded message, it is crucial for +not getting stuck to use a timeout for each packet: when the timeout is reached, +an IP packet with an increased ttl is sent and an unknown for the ttl is +printed (see the fifth hop in the example above).

The packets send out are conventionally UDP packets without payload. From a +development perspective, one question is how to correlate the ICMP packet +with the sent UDP packet. Conveniently, ICMP packets contain the IP header and +the first eight bytes of the next protocol - the UDP header containing source +port, destination port, checksum, and payload length (each fields of size two +bytes). This means when we record the outgoing ports together with the sent +timestamp, and correlate the later received ICMP packet to the sent packet. +Great.

But as a functional programmer, let's figure whether we can abolish the +(globally shared) state. Since the ICMP packet contains the original IP +header and the first eight bytes of the UDP header, this is where we will +embed data. As described above, the data is the sent timestamp and the value +of the ttl field. For the latter, we can arbitrarily restrict it to 31 (5 bits). +For the timestamp, it is mainly a question about precision and maximum expected +round trip time. Taking the source and destination port are 32 bits, using 5 for +ttl, remaining are 27 bits (an unsigned value up to 134217727). Looking at the +decimal representation, 1 second is likely too small, 13 seconds are sufficient +for the round trip time measurement. This implies our precision is 100ns, by +counting the digits.

Finally to the code. First we need forth and back conversions between ports +and ttl, timestamp:

(* takes a time-to-live (int) and timestamp (int64, nanoseconda), encodes them
+   into 16 bit source port and 16 bit destination port:
+   - the timestamp precision is 100ns (thus, it is divided by 100)
+   - use the bits 27-11 of the timestamp as source port
+   - use the bits 11-0 as destination port, and 5 bits of the ttl
+*)
+let ports_of_ttl_ts ttl ts =
+  let ts = Int64.div ts 100L in
+  let src_port = 0xffff land (Int64.(to_int (shift_right ts 11)))
+  and dst_port = 0xffe0 land (Int64.(to_int (shift_left ts 5))) lor (0x001f land ttl)
+  in
+  src_port, dst_port
+
+(* inverse operation of ports_of_ttl_ts for the range (src_port and dst_port
+   are 16 bit values) *)
+let ttl_ts_of_ports src_port dst_port =
+  let ttl = 0x001f land dst_port in
+  let ts =
+    let low = Int64.of_int (dst_port lsr 5)
+    and high = Int64.(shift_left (of_int src_port) 11)
+    in
+    Int64.add low high
+  in
+  let ts = Int64.mul ts 100L in
+  ttl, ts
+

They should be inverse over the range of valid input: ports are 16 bit numbers, +ttl expected to be at most 31, ts a int64 expressed in nanoseconds.

Related is the function to print out one hop and round trip measurement:

(* write a log line of a hop: the number, IP address, and round trip time *)
+let log_one now ttl sent ip =
+  let now = Int64.(mul (logand (div now 100L) 0x7FFFFFFL) 100L) in
+  let duration = Mtime.Span.of_uint64_ns (Int64.sub now sent) in
+  Logs.info (fun m -> m "%2d  %a  %a" ttl Ipaddr.V4.pp ip Mtime.Span.pp duration)
+

The most logic is when a ICMP packet is received:

module Icmp = struct
+  type t = {
+    send : int -> unit Lwt.t ;
+    log : int -> int64 -> Ipaddr.V4.t -> unit ;
+    task_done : unit Lwt.u ;
+  }
+
+  let connect send log task_done =
+    let t = { send ; log ; task_done } in
+    Lwt.return t
+
+  (* This is called for each received ICMP packet. *)
+  let input t ~src ~dst buf =
+    let open Icmpv4_packet in
+    (* Decode the received buffer (the IP header has been cut off already). *)
+    match Unmarshal.of_cstruct buf with
+    | Error s ->
+      Lwt.fail_with (Fmt.strf "ICMP: error parsing message from %a: %s" Ipaddr.V4.pp src s)
+    | Ok (message, payload) ->
+      let open Icmpv4_wire in
+      (* There are two interesting cases: Time exceeded (-> send next packet),
+         and Destination (port) unreachable (-> we reached the final host and can exit) *)
+      match message.ty with
+      | Time_exceeded ->
+        (* Decode the payload, which should be an IPv4 header and a protocol header *)
+        begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
+          | Ok (pkt, off) when
+              (* Ensure this packet matches our sent packet: the protocol is UDP
+                 and the destination address is the host we're tracing *)
+              pkt.Ipv4_packet.proto = Ipv4_packet.Marshal.protocol_to_int `UDP &&
+              Ipaddr.V4.compare pkt.Ipv4_packet.dst (Key_gen.host ()) = 0 ->
+            let src_port = Cstruct.BE.get_uint16 payload off
+            and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
+            in
+            (* Retrieve ttl and sent timestamp, encoded in the source port and
+               destination port of the UDP packet we sent, and received back as
+               ICMP payload. *)
+            let ttl, sent = ttl_ts_of_ports src_port dst_port in
+            (* Log this hop. *)
+            t.log ttl sent src;
+            (* Sent out the next UDP packet with an increased ttl. *)
+            let ttl' = succ ttl in
+            Logs.debug (fun m -> m "ICMP time exceeded from %a to %a, now sending with ttl %d"
+                           Ipaddr.V4.pp src Ipaddr.V4.pp dst ttl');
+            t.send ttl'
+          | Ok (pkt, _) ->
+            (* Some stray ICMP packet. *)
+            Logs.debug (fun m -> m "unsolicited time exceeded from %a to %a (proto %X dst %a)"
+                           Ipaddr.V4.pp src Ipaddr.V4.pp dst pkt.Ipv4_packet.proto Ipaddr.V4.pp pkt.Ipv4_packet.dst);
+            Lwt.return_unit
+          | Error e ->
+            (* Decoding error. *)
+            Logs.warn (fun m -> m "couldn't parse ICMP time exceeded payload (IPv4) (%a -> %a) %s"
+                          Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
+            Lwt.return_unit
+        end
+      | Destination_unreachable when Ipaddr.V4.compare src (Key_gen.host ()) = 0 ->
+        (* We reached the final host, and the destination port was not listened to *)
+        begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
+          | Ok (_, off) ->
+            let src_port = Cstruct.BE.get_uint16 payload off
+            and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
+            in
+            (* Retrieve ttl and sent timestamp. *)
+            let ttl, sent = ttl_ts_of_ports src_port dst_port in
+            (* Log the final hop. *)
+            t.log ttl sent src;
+            (* Wakeup the waiter task to exit the unikernel. *)
+            Lwt.wakeup t.task_done ();
+            Lwt.return_unit
+          | Error e ->
+            (* Decoding error. *)
+            Logs.warn (fun m -> m "couldn't parse ICMP unreachable payload (IPv4) (%a -> %a) %s"
+                          Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
+            Lwt.return_unit
+        end
+      | ty ->
+        Logs.debug (fun m -> m "ICMP unknown ty %s from %a to %a: %a"
+                       (ty_to_string ty) Ipaddr.V4.pp src Ipaddr.V4.pp dst
+                       Cstruct.hexdump_pp payload);
+        Lwt.return_unit
+end
+

Now, the remaining main unikernel is the module Main:

module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.S) (N : Mirage_net.S) = struct
+  module ETH = Ethernet.Make(N)
+  module ARP = Arp.Make(ETH)(Time)
+  module IPV4 = Static_ipv4.Make(R)(M)(ETH)(ARP)
+  module UDP = Udp.Make(IPV4)(R)
+
+  (* Global mutable state: the timeout task for a sent packet. *)
+  let to_cancel = ref None
+
+  (* Send a single packet with the given time to live. *)
+  let rec send_udp udp ttl =
+    (* This is called by the ICMP handler which successfully received a
+       time exceeded, thus we cancel the timeout task. *)
+    (match !to_cancel with
+     | None -> ()
+     | Some t -> Lwt.cancel t ; to_cancel := None);
+    (* Our hop limit is 31 - 5 bit - should be sufficient for most networks. *)
+    if ttl > 31 then
+      Lwt.return_unit
+    else
+      (* Create a timeout task which:
+         - sleeps for --timeout interval
+         - logs an unknown hop
+         - sends another packet with increased ttl
+      *)
+      let cancel =
+        Lwt.catch (fun () ->
+            Time.sleep_ns (Duration.of_ms (Key_gen.timeout ())) >>= fun () ->
+            Logs.info (fun m -> m "%2d  *" ttl);
+            send_udp udp (succ ttl))
+          (function Lwt.Canceled -> Lwt.return_unit | exc -> Lwt.fail exc)
+      in
+      (* Assign this timeout task. *)
+      to_cancel := Some cancel;
+      (* Figure out which source and destination port to use, based on ttl
+         and current timestamp. *)
+      let src_port, dst_port = ports_of_ttl_ts ttl (M.elapsed_ns ()) in
+      (* Send packet via UDP. *)
+      UDP.write ~ttl ~src_port ~dst:(Key_gen.host ()) ~dst_port udp Cstruct.empty >>= function
+      | Ok () -> Lwt.return_unit
+      | Error e -> Lwt.fail_with (Fmt.strf "while sending udp frame %a" UDP.pp_error e)
+
+  (* The main unikernel entry point. *)
+  let start () () () net =
+    let cidr = Key_gen.ipv4 ()
+    and gateway = Key_gen.ipv4_gateway ()
+    in
+    let log_one = fun port ip -> log_one (M.elapsed_ns ()) port ip
+    (* Create a task to wait on and a waiter to wakeup. *)
+    and t, w = Lwt.task ()
+    in
+    (* Setup network stack: ethernet, ARP, IPv4, UDP, and ICMP. *)
+    ETH.connect net >>= fun eth ->
+    ARP.connect eth >>= fun arp ->
+    IPV4.connect ~cidr ~gateway eth arp >>= fun ip ->
+    UDP.connect ip >>= fun udp ->
+    let send = send_udp udp in
+    Icmp.connect send log_one w >>= fun icmp ->
+
+    (* The callback cascade for an incoming network packet. *)
+    let ethif_listener =
+      ETH.input
+        ~arpv4:(ARP.input arp)
+        ~ipv4:(
+          IPV4.input
+            ~tcp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
+            ~udp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
+            ~default:(fun ~proto ~src ~dst buf ->
+                match proto with
+                | 1 -> Icmp.input icmp ~src ~dst buf
+                | _ -> Lwt.return_unit)
+            ip)
+        ~ipv6:(fun _ -> Lwt.return_unit)
+        eth
+    in
+    (* Start the callback in a separate asynchronous task. *)
+    Lwt.async (fun () ->
+        N.listen net ~header_size:Ethernet_wire.sizeof_ethernet ethif_listener >|= function
+        | Ok () -> ()
+        | Error e -> Logs.err (fun m -> m "netif error %a" N.pp_error e));
+    (* Send the initial UDP packet with a ttl of 1. This entails the domino
+       effect to receive ICMP packets, send out another UDP packet with ttl
+       increased by one, etc. - until a destination unreachable is received,
+       or the hop limit is reached. *)
+    send 1 >>= fun () ->
+    t
+end
+

The configuration (config.ml) for this unikernel is as follows:

open Mirage
+
+let host =
+  let doc = Key.Arg.info ~doc:"The host to trace." ["host"] in
+  Key.(create "host" Arg.(opt ipv4_address (Ipaddr.V4.of_string_exn "141.1.1.1") doc))
+
+let timeout =
+  let doc = Key.Arg.info ~doc:"Timeout (in millisecond)" ["timeout"] in
+  Key.(create "timeout" Arg.(opt int 1000 doc))
+
+let ipv4 =
+  let doc = Key.Arg.info ~doc:"IPv4 address" ["ipv4"] in
+  Key.(create "ipv4" Arg.(required ipv4 doc))
+
+let ipv4_gateway =
+  let doc = Key.Arg.info ~doc:"IPv4 gateway" ["ipv4-gateway"] in
+  Key.(create "ipv4-gateway" Arg.(required ipv4_address doc))
+
+let main =
+  let packages = [
+    package ~sublibs:["ipv4"; "udp"; "icmpv4"] "tcpip";
+    package "ethernet";
+    package "arp-mirage";
+    package "mirage-protocols";
+    package "mtime";
+  ] in
+  foreign
+    ~keys:[Key.abstract ipv4 ; Key.abstract ipv4_gateway ; Key.abstract host ; Key.abstract timeout]
+    ~packages
+    "Unikernel.Main"
+    (random @-> mclock @-> time @-> network @-> job)
+
+let () =
+  register "traceroute"
+    [ main $ default_random $ default_monotonic_clock $ default_time $ default_network ]
+

And voila, that's all the code. If you copy it together (or download the two +files from the GitHub repository), +and have OCaml, opam, and mirage (>= 3.8.0) installed, +you should be able to:

$ mirage configure -t hvt
+$ make depend
+$ make
+$ solo5-hvt --net:service=tap0 -- traceroute.hvt ...
+... get the output shown at top ...
+

Enhancements may be to use a different protocol (TCP? or any other protocol ID (may be used to encode more information), encode data into IPv4 ID, or the full 8 bytes of the upper protocol), encrypt/authenticate the data transmitted (and verify it has not been tampered with in the ICMP reply), improve error handling and recovery, send multiple packets for improved round trip time measurements, ...

If you develop enhancements you'd like to share, please sent a pull request to the git repository.

Motivation for this traceroute unikernel was while talking with Aaron and Paul, who contributed several patches to the IP stack which pass the ttl through.

If you want to support our work on MirageOS unikernels, please donate to robur. I'm interested in feedback, either via twitter, hannesm@mastodon.social or via eMail.

Deploying authoritative OCaml-DNS servers as MirageOS unikernels

hannes — 2019-12-23T21:30:53-00:00

Goal

Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as nqsb.io and robur.coop. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as µDNS.

Prerequisites

You need to own a domain, and be able to delegate the name service to your own servers. +You also need two spare public IPv4 addresses (in different /24 networks) for your name servers. +A git server or remote repository reachable via git over ssh. +Servers which support solo5 guests, and have the corresponding tender installed. +A computer with opam (>= 2.0.0) installed.

Data preparation

Figure out a way to get the DNS entries of your domain in a "master file format", i.e. what bind uses.

This is a master file for the mirage domain, defining $ORIGIN to avoid typing the domain name after each hostname (use @ if you need the domain name only; if you need to refer to a hostname in a different domain end it with a dot (.), i.e. ns2.foo.com.). The default time to live $TTL is an hour (3600 seconds). +The zone contains a start of authority (SOA) record containing the nameserver, hostmaster, serial, refresh, retry, expiry, and minimum. +Also, a single name server (NS) record ns1 is specified with an accompanying address (A) records pointing to their IPv4 address.

git-repo> cat mirage
+$ORIGIN mirage.
+$TTL 3600
+@	SOA	ns1	hostmaster	1	86400	7200	1048576	3600
+@	NS	ns1
+ns1     A       127.0.0.1
+www	A	1.1.1.1
+git-repo> git add mirage && git commit -m initial && git push
+

Installation

On your development machine, you need to install various OCaml packages. You don't need privileged access if common tools (C compiler, make, libgmp) are already installed. You have opam installed.

Let's create a fresh switch for the DNS journey:

$ opam init
+$ opam update
+$ opam switch create udns 4.14.1
+# waiting a bit, a fresh OCaml compiler is getting bootstrapped
+$ eval `opam env` #sets some environment variables
+

The last command set environment variables in your current shell session, please use the same shell for the commands following (or run eval $(opam env) in another shell and proceed in there - the output of opam switch sohuld point to udns).

Validation of our zonefile

First let's check that OCaml-DNS can parse our zonefile:

$ opam install dns-cli #installs ~/.opam/udns/bin/ozone and other binaries
+$ ozone <git-repo>/mirage # see ozone --help
+successfully checked zone
+

Great. Error reporting is not great, but line numbers are indicated (ozone: zone parse problem at line 3: syntax error), lexer and parser are lex/yacc style (PRs welcome).

FWIW, ozone accepts --old <filename> to check whether an update from the old zone to the new is fine. This can be used as pre-commit hook in your git repository to avoid bad parse states in your name servers.

Getting the primary up

The next step is to compile the primary server and run it to serve the domain data. Since the git-via-ssh client is not yet released, we need to add a custom opam repository to this switch.

# get the `mirage` application via opam
+$ opam install lwt mirage
+
+# get the source code of the unikernels
+$ git clone https://github.com/roburio/dns-primary-git.git
+$ cd dns-primary-git
+
+# let's build the server first as unix application
+$ mirage configure #--no-depext if you have all system dependencies
+$ make
+
+# run it
+$ dist/primary-git
+# starts a unix process which clones https://github.com/roburio/udns.git
+# attempts to parse the data as zone files, and fails on parse error
+$ dist/primary-git --remote=https://my-public-git-repository
+# this should fail with ENOACCESS since the DNS server tries to listen on port 53
+
+# which requires a privileged user, i.e. su, sudo or doas
+$ sudo dist/primary-git --remote=https://my-public-git-repository
+# leave it running, run the following programs in a different shell
+
+# test it
+$ host ns1.mirage 127.0.0.1
+ns1.mirage has address 127.0.0.1
+$ dig any mirage @127.0.0.1
+# a DNS packet printout with all records available for mirage
+

That's exciting, the DNS server serving answers from a remote git repository.

Securing the git access with ssh

Let's authenticate the access by using ssh, so we feel ready to push data there as well. The primary-git unikernel already includes an experimental ssh client, all we need to do is setting up credentials - in the following a RSA keypair and the server fingerprint.

# collect the RSA host key fingerprint
+$ ssh-keyscan <git-server> > /tmp/git-server-public-keys
+$ ssh-keygen -l -E sha256 -f /tmp/git-server-public-keys | grep ED25519
+256 SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 <git-server> (ED25519)
+# we're interested in the SHA256:yyy only
+
+# generate a ssh keypair
+$ awa_gen_key --keytype ed25510 # installed by the make step above in ~/.opam/udns/bin
+private key: ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w=
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyliWbwWWXBc1+DRIzReLQ4UFiVGXJ6Paw1Jts+XQte awa@awa.local
+# please run your own awa_gen_key, don't use the numbers above
+

The public key needs is in standard OpenSSH format and needs to be added to the list of accepted keys on your server - the exact steps depend on your git server, if you're running your own with gitosis, add it as new public key file and grant that key access to the data repository. If you use gitlab or github, you may want to create a new user account and with the generated key.

The private key is not displayed, but only the seed required to re-generate it, when using the same random number generator, in our case fortuna implemented by mirage-crypto - used by both awa_gen_key and primary-git. The seed is provided as command-line argument while starting primary-git:

# execute with git over ssh, authenticator from ssh-keyscan, seed from awa_gen_key
+$ ./primary-git --authenticator=SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 --ssh-key=ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w= --remote=git@<git-server>:repo-name.git
+# started up, you can try the host and dig commands from above if you like
+

To wrap up, we now have a primary authoritative name server for our zone running as Unix process, which clones a remote git repository via ssh on startup and then serves it.

Authenticated data updates

Our remote git repository is the source of truth, if you need to add a DNS entry to the zone, you git pull, edit the zone file, remember to increase the serial in the SOA line, run ozone, git commit and push to the repository.

So, the primary-git needs to be informed of git pushes. This requires a communication channel from the git server (or somewhere else, e.g. your laptop) to the DNS server. I prefer in-protocol solutions over adding yet another protocol stack, no way my DNS server will talk HTTP REST.

The DNS protocol has an extension for notifications of zone changes (as a DNS packet), usually used between the primary and secondary servers. The primary-git accepts these notify requests (i.e. bends the standard slightly), and upon receival pulls the remote git repository, and serves the fresh zone files. Since a git pull may be rather excessive in terms of CPU cycles and network bandwidth, only authenticated notifications are accepted.

The DNS protocol specifies in another extension authentication (DNS TSIG) with transaction signatures on DNS packets including a timestamp and fudge to avoid replay attacks. As key material hmac secrets distribued to both the communication endpoints are used.

To recap, the primary server is configured with command line parameters (for remote repository url and ssh credentials), and serves data from a zonefile. If the secrets would be provided via command line, a restart would be necessary for adding and removing keys. If put into the zonefile, they would be publicly served on request. So instead, we'll use another file, still in zone file format, in the top-level domain _keys, i.e. the mirage._keys file contains keys for the mirage zone. All files ending in ._keys are parsed with the normal parser, but put into an authentication store instead of the domain data store, which is served publically.

For encoding hmac secrets into DNS zone file format, the DNSKEY format is used (designed for DNSsec). The bind software comes with dnssec-keygen and tsig-keygen to generate DNSKEY output: flags is 0, protocol is 3, and algorithm identifier for SHA256 is 163 (SHA384 164, SHA512 165). This is reused by the OCaml DNS library. The key material itself is base64 encoded.

Access control and naming of keys follows the DNS domain name hierarchy - a key has the form name._operation.domain, and has access granted to domain and all subdomains of it. Two operations are supported: update and transfer. In the future there may be a dedicated notify operation, for now we'll use update. The name part is ignored for the update operation.

Since we now embedd secret information in the git repository, it is a good idea to restrict access to it, i.e. make it private and not publicly cloneable or viewable. Let's generate a first hmac secret and send a notify:

$ dd if=/dev/random bs=1 count=32 | b64encode -
+begin-base64 644 -
+kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
+====
+[..]
+git-repo> echo "personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=" > mirage._keys
+git-repo> git add mirage._keys && git commit -m "add hmac secret" && git push
+
+# now we need to restart the primary git to get the git repository with the key
+$ ./primary-git --ssh-key=... # arguments from above, remote git, host key fingerprint, private key seed
+
+# now test that a notify results in a git pull
+$ onotify 127.0.0.1 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
+# onotify was installed by dns-cli in ~/.opam/udns/bin/onotify, see --help for options
+# further changes to the hmac secrets don't require a restart anymore, a notify packet is sufficient :D
+

Ok, this onotify command line could be setup as a git post-commit hook, or run manually after each manual git push.

Secondary

It's time to figure out how to integrate the secondary name server. An already existing bind or something else that accepts notifications and issues zone transfers with hmac-sha256 secrets should work out of the box. If you encounter interoperability issues, please get in touch with me.

The secondary unikernel is available from another git repository:

# get the secondary sources
+$ git clone https://github.com/roburio/dns-secondary.git
+$ cd dns-secondary
+

It's only command line argument is a list of hmac secrets used for authenticating that the received data originates from the primary server. Data is initially transferred by a full zone transfer (AXFR), later updates (upon refresh timer or notify request sent by the primary) use incremental (IXFR). Zone transfer requests and data are authenticated with transaction signatures again.

Convenience by OCaml DNS is that transfer key names matter, and are of the form .._transfer.domain, i.e. 1.1.1.1.2.2.2.2._transfer.mirage if the primary server is 1.1.1.1, and the secondary 2.2.2.2. Encoding the IP address in the name allows both parties to start the communication: the secondary starts by requesting a SOA for all domains for which keys are provided on command line, and if an authoritative SOA answer is received, the AXFR is triggered. The primary server emits notification requests on startup and then on every zone change (i.e. via git pull) to all secondary IP addresses of transfer keys present for the specific zone in addition to the notifications to the NS records in the zone.

$ mirage configure
+$ make
+$ ./dist/secondary
+

IP addresses and routing

Both primary and secondary serve the data on the DNS port (53) on UDP and TCP. To run both on the same machine and bind them to different IP addresses, we'll use a layer 2 network (ethernet frames) with a host system software switch (bridge interface service), the unikernels as virtual machines (or seccomp-sandboxed) via the solo5 backend. Using xen is possible as well. As IP address range we'll use 10.0.42.0/24, and the host system uses the 10.0.42.1.

The primary git needs connectivity to the remote git repository, thus on a laptop in a private network we need network address translation (NAT) from the bridge where the unikernels speak to the Internet where the git repository resides.

# on FreeBSD:
+# configure NAT with pf, you need to have forwarding enabled
+$ sysctl net.inet.ip.forwarding: 1
+$ echo 'nat pass on wlan0 inet from 10.0.42.0/24 to any -> (wlan0)' >> /etc/pf.conf
+$ service pf restart
+
+# make tap interfaces UP on open()
+$ sysctl net.link.tap.up_on_open: 1
+
+# bridge creation, naming, and IP setup
+$ ifconfig bridge create
+bridge0
+$ ifconfig bridge0 name service
+$ ifconfig bridge0 10.0.42.1/24
+
+# two tap interfaces for our unikernels
+$ ifconfig tap create
+tap0
+$ ifconfig tap create
+tap1
+# add them to the bridge
+$ ifconfig service addm tap0 addm tap1
+

Primary and secondary setup

Let's update our zone slightly to reflect the IP changes.

git-repo> cat mirage
+$ORIGIN mirage.
+$TTL 3600
+@	SOA	ns1	hostmaster	2	86400	7200	1048576	3600
+@	NS	ns1
+@	NS	ns2
+ns1     A       10.0.42.2
+ns2	A	10.0.42.3
+
+# we also need an additional transfer key
+git-repo> cat mirage._keys
+personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
+10.0.42.2.10.0.42.3._transfer.mirage. DNSKEY 0 3 163 cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20=
+git-repo> git commit -m "updates" . && git push
+

Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see other targets for further information).

# back to primary
+$ cd ../dns-primary-git
+$ mirage configure -t hvt # or e.g. -t spt (and solo5-spt below)
+# installs backend-specific opam packages, recompiles some
+$ make
+[...]
+$ solo5-hvt --net:service=tap0 -- primary_git.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --seed=.. --authenticator=.. --remote=...
+# should now run as a virtual machine (kvm, bhyve), and clone the git repository
+$ dig any mirage @10.0.42.2
+# should reply with the SOA and NS records, and also the name server address records in the additional section
+
+# secondary
+$ cd ../dns-secondary
+$ mirage configure -t hvt
+$ make
+$ solo5-hvt --net:service=tap1 -- secondary.hvt --ipv4=10.0.42.3/24 --keys=10.0.42.2.10.0.42.3._transfer.mirage:SHA256:cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20=
+# an ipv4-gateway is not needed in this setup, but in real deployment later
+# it should start up and transfer the mirage zone from the primary
+
+$ dig any mirage @10.0.42.3
+# should now output the same information as from 10.0.42.2
+
+# testing an update and propagation
+# edit mirage zone, add a new record and increment the serial number
+git-repo> echo "foo A 127.0.0.1" >> mirage
+git-repo> vi mirage <- increment serial
+git-repo> git commit -m 'add foo' . && git push
+$ onotify 10.0.42.2 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
+
+# now check that it worked
+$ dig foo.mirage @10.0.42.2 # primary
+$ dig foo.mirage @10.0.42.3 # secondary got notified and transferred the zone
+

You can also check the behaviour when restarting either of the VMs, whenever the primary is available the zone is synchronised. If the primary is down, the secondary still serves the zone. When the secondary is started while the primary is down, it won't serve any data until the primary is online (the secondary polls periodically, the primary sends notifies on startup).

Dynamic data updates via DNS, pushed to git

DNS is a rich protocol, and it also has builtin updates that are supported by OCaml DNS, again authenticated with hmac-sha256 and shared secrets. Bind provides the command-line utility nsupdate to send these update packets, a simple oupdate unix utility is available as well (i.e. for integration of dynamic DNS clients). You know the drill, add a shared secret to the primary, git push, notify the primary, and voila we can dynamically in-protocol update. An update received by the primary via this way will trigger a git push to the remote git repository, and notifications to the secondary servers as described above.

# being lazy, I reuse the key above
+$ oupdate 10.0.42.2 personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg= my-other.mirage 1.2.3.4
+
+# let's observe the remote git
+git-repo> git pull
+# there should be a new commit generated by the primary
+git-repo> git log
+
+# test it, should return 1.2.3.4
+$ dig my-other.mirage @10.0.42.2
+$ dig my-other.mirage @10.0.42.3
+

So we can deploy further oupdate (or nsupdate) clients, distribute hmac secrets, and have the DNS zone updated. The source of truth is still the git repository, where the primary-git pushes to. Merge conflicts and timing of pushes is not yet dealt with. They are unlikely to happen since the primary is notified on pushes and should have up-to-date data in storage. Sorry, I'm unsure about the error semantics, try it yourself.

Let's encrypt!

Let's encrypt is a certificate authority (CA), which certificate is shipped as trust anchor in web browsers. They specified a protocol for automated certificate management environment (ACME), used to get X509 certificates for your services. In the protocol, a certificate signing request (publickey and hostname) is sent to let's encrypt servers, which sends a challenge to proof the ownership of the hostnames. One widely-used way to solve this challenge is running a web server, another is to serve it as text record from the authoritative DNS server.

Since I avoid persistent storage when possible, and also don't want to integrate a HTTP client stack in the primary server, I developed a third unikernel that acts as (hidden) secondary server, performs the tedious HTTP communication with let's encrypt servers, and stores all data in the public DNS zone.

For encoding of certificates, the DANE working group specified TLSA records in DNS. They are quadruples of usage, selector, matching type, and ASN.1 DER-encoded material. We set usage to 3 (domain-issued certificate), matching type to 0 (no hash), and selector to 0 (full certificate) or 255 (private usage) for certificate signing requests. The interaction is as follows:

Primary, secondary, and let's encrypt unikernels are running +
A service (ocertify, unikernels/certificate, or the dns-certify.mirage library) demands a TLS certificate, and has a hmac-secret for the primary DNS +
The service generates a certificate signing request with the desired hostname(s), and performs an nsupdate with TLSA 255 +
The primary accepts the update, pushes the new zone to git, and sends notifies to secondary and let's encrypt unikernels which (incrementally) transfer the zone +
The let's encrypt unikernel notices while transferring the zone a signing request without a certificate, starts HTTP interaction with let's encrypt +
The let's encrypt unikernel solves the challenge, sends the response as update of a TXT record to the primary nameserver +
The primary pushes the TXT record to git, and notifies secondaries (which transfer the zone) +
The let's encrypt servers request the TXT record from either or both authoritative name servers +
The let's encrypt unikernel polls for the issued certificate and send an update to the primary TLSA 0 +
The primary pushes the certificate to git, notifies secondaries (which transfer the zone) +
The service polls TLSA records for the hostname, and use it upon retrieval +

Note that neither the signing request nor the certificate contain private key material, thus it is fine to serve them publically. Please also note, that the service polls for the certificate for the hostname in DNS, which is valid (start and end date) certificate and uses the same public key, this certificate is used and steps 3-10 are not executed.

The let's encrypt unikernel does not serve anything, it is a reactive system which acts upon notification from the primary. Thus, it can be executed in a private address space (with a NAT). Since the OCaml DNS server stack needs to push notifications to it, it preserves all incoming signed SOA requests as candidates for notifications on update. The let's encrypt unikernel ensures to always have a connection to the primary to receive notifications.

# getting let's encrypt up and running
+$ cd ..
+$ git clone https://github.com/roburio/dns-letsencrypt-secondary.git
+$ cd dns-letsencrypt-secondary
+$ mirage configure -t hvt
+$ make
+
+# run it
+$ solo5-hvt --net:service=tap2 -- letsencrypt.hvt --keys=...
+
+# test it
+$ ocertify 10.0.42.2 foo.mirage
+

For actual testing with let's encrypt servers you need to have the primary and secondary deployed on your remote hosts, and your domain needs to be delegated to these servers. Good luck. And ensure you have backup your git repository.

As fine print, while this tutorial was about the mirage zone, you can stick any number of zones into the git repository. If you use a _keys file (without any domain prefix), you can configure hmac secrets for all zones, i.e. something to use in your let's encrypt unikernel and secondary unikernel. Dynamic addition of zones is supported, just create a new zonefile and notify the primary, the secondary will be notified and pick it up. The primary responds to a signed SOA for the root zone (i.e. requested by the secondary) with the SOA response (not authoritative), and additionally notifications for all domains of the primary.

Conclusion and thanks

This tutorial presented how to use the OCaml DNS based unikernels to run authoritative name servers for your domain, using a git repository as the source of truth, dynamic authenticated updates, and let's encrypt certificate issuing.

There are further steps to take, such as monitoring (mirage configure --monitoring), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.

I'd like to thank all people involved in this software stack, without other key components, including git, mirage-crypto, awa-ssh, solo5, mirage, ocaml-letsencrypt, and more.

If you want to support our work on MirageOS unikernels, please donate to robur. I'm interested in feedback, either via twitter, hannesm@mastodon.social or via eMail.

Reproducible MirageOS unikernel builds

hannes — 2019-12-16T18:29:30-00:00

Reproducible builds summit

I'm just back from the Reproducible builds summit 2019. In 2018, several people developing OCaml and opam and MirageOS, attended the Reproducible builds summit in Paris. The notes from last year on opam reproducibility and MirageOS reproducibility are online. After last years workshop, Raja started developing the opam reproducibilty builder orb, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.

Goal

Stepping back a bit, first look on the goal of reproducible builds: when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check bit-wise identity of binaries, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.

A checklist of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed disorderfs for testing reproducibility and diffoscope for comparing binaries with file-dependent readers, falling back to objdump and hexdump. A giant test infrastructure with lots of variations between the builds, mostly using Debian, has been setup over the years.

Reproducibility is a precondition for trustworthy binaries. See why does it matter. If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. I will only ever publish binaries if they are reproducible.

My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of MirageOS unikernels -- a unikernel is a statically linked ELF binary to be run as Unix process or virtual machine. MirageOS heavily uses OCaml and opam, the OCaml package manager, and is an opam package itself. Thus, checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package.

Reproducible builds with opam

Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses a community repository where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.

Conceptually there should be two tools, the initial builder, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a rebuilder, which imports the export, conducts a build, and outputs the hashes of the produced binaries.

Opam has the concept of a switch, which is an environment where a package set is installed. Switches are independent of each other, and can already be exported and imported. Unfortunately the export is incomplete: if a package includes additional patches as part of the repository -- sometimes needed for fixing releases where the actual author or maintainer of a package responds slowly -- these package neither the patches end up in the export. Also, if a package is pinned to a git branch, the branch appears in the export, but this may change over time by pushing more commits or even force-pushing to that branch. In PR #4040 (under discussion and review), also developed during the summit, I propose to embed the additional files as base64 encoded values in the opam file. To solve the latter issue, I modified the export mechanism to embed the git commit hash (PR #4055), and avoid sources from a local directory and which do not have a checksum.

So the opam export contains the information required to gather the exact same sources and build instructions of the opam packages. If the opam repository would be self-contained (i.e. not depend on any other tools), this would be sufficient. But opam does not run in thin air, it requires some system utilities such as /bin/sh, sed, a GNU make, commonly git, a C compiler, a linker, an assembler. Since opam is available on various operating systems, the plugin depext handles host system dependencies, e.g. if your opam package requires gmp to be installed, this requires slightly different names depending on host system or distribution, take a look at conf-gmp. This also means, opam has rather good information about both the opam dependencies and the host system dependencies for each package. Please note that the host system packages used during compilation are not yet recorded (i.e. which gmp package was installed and used during the build, only that a gmp package has to be installed). The base utilities mentioned above (C compiler, linker, shell) are also not recorded yet.

Operating system information available in opam (such as architecture, distribution, version), which in some cases maps to exact base utilities, is recorded in the build-environment, a separate artifact. The environment variable SOURCE_DATE_EPOCH, used for communicating the same timestamp when software is required to record a timestamp into the resulting binary, is also captured in the build environment.

Additional environment variables may be captured or used by opam packages to produce different output. To avoid this, both the initial builder and the rebuilder are run with minimal environment variables: only PATH (normalised to a whitelist of /bin, /usr/bin, /usr/local/bin and /opt/bin) and HOME are defined. Missing information at the moment includes CPU features: some libraries (gmp?, nocrypto) emit different code depending on the CPU feature.

Tooling

TL;DR: A build builds an opam package, and outputs .opam-switch, .build-hashes.N, and .build-environment.N. A rebuild uses these artifacts as input, builds the package and outputs another .build-hashes.M and .build-environment.M.

The command-line utility orb can be installed and used:

$ opam pin add orb git+https://github.com/hannesm/orb.git#active
+$ orb build --twice --keep-build-dir --diffoscope <your-favourite-opam-package>
+

It provides two subcommands build and rebuild. The build command takes a list of local opam --repos where to take opam packages from (defaults to default), a compiler (either a variant --compiler=4.09.0+flambda, a version --compiler=4.06.0, or a pin to a local development version --compiler-pin=~/ocaml), and optionally an existing switch --use-switch. It creates a switch, builds the packages, and emits the opam export, hashes of all files installed by these packages, and the build environment. The flags --keep-build retains the build products, opam's --keep-build-dir in addition temporary build products and generated source code. If --twice is provided, a rebuild (described next) is executed after the initial build.

The rebuild command takes a directory with the opam export and build environment to build the opam package. It first compares the build-environment with the host system, sets the SOURCE_DATE_EPOCH and switch location accordingly and executes the import. Once the build is finished, it compares the hashes of the resulting files with the previous run. On divergence, if build directories were kept in the previous build, and if diffoscope is available and --diffoscope was provided, diffoscope is run on the diverging files. If --keep-build-dir was provided as well, diff -ur can be used to compare the temporary build and sources, including build logs.

The builds are run in parallel, as opam does, this parallelism does not lead to different binaries in my experiments.

Results and discussion

All MirageOS unikernels I have deployed are reproducible \o/. Also, several binaries such as orb itself, opam, solo5-hvt, and all albatross utilities are reproducible.

The unikernel range from hello world, web servers (e.g. this blog, getting its data on startup via a git clone to memory), authoritative DNS servers, CalDAV server. They vary in size between 79 and 200 opam packages, resulting in 2MB - 16MB big ELF binaries (including debug symbols). The unikernel opam repository contains some reproducible unikernels used for testing. Some work-in-progress enhancements are needed to achieve this:

At the moment, the opam package of a MirageOS unikernel is automatically generated by mirage configure, but only used for tracking opam dependencies. I worked on mirage PR #1022 to extend the generated opam package with build and install instructions.

As mentioned above, if locale is set, ocamlgraph needs to be patched to emit a (locale-dependent) timestamp.

The OCaml program crunch embeds a subdirectory as OCaml code into a binary, which we use in MirageOS quite regularly for static assets, etc. This plays in several ways into reproducibility: on the one hand, it needs a timestamp for its last_modified functionality (and adheres since June 2018 to the SOURCE_DATE_EPOCH spec, thanks to Xavier Clerc). On the other hand, it used before version 3.2.0 (released Dec 14th) hashtables for storing the file contents, where iteration is not deterministic (the insertion is not sorted), fixed in PR #51 by using a Map instead.

In functoria, a tool used to configure MirageOS devices and their dependencies, can emit a list of opam packages which were required to build the unikernel. This uses opam list --required-by --installed --rec <pkgs>, which uses the cudf graph (thanks to Raja for explanation), that is during the rebuild dropping some packages. The PR #189 avoids by not using the --rec argument, but manually computing the fixpoint.

Certainly, the choice of environment variables, and whether to vary them (as debian does) or to not define them (or normalise) while building, is arguably. Since MirageOS does neither support time zone nor internationalisation, there is no need to prematurely solving this issue. On related note, even with different locale settings, MirageOS unikernels are reproducible apart from an issue in ocamlgraph #90 embedding the output of date, which is different depending on LANG and locale (LC_*) settings.

Prior art in reproducible MirageOS unikernels is the mirage-qubes-firewall. Since early 2017 it is reproducible. Their approach is different by building in a docker container with the opam repository pinned to an exact git commit.

Further work

I only tested a certain subset of opam packages and MirageOS unikernels, mainly on a single machine (my laptop) running FreeBSD, and am happy if others will test reproducibility of their OCaml programs with the tools provided. There could as well be CI machines rebuilding opam packages and reporting results to a central repository. I'm pretty sure there are more reproducibility issues in the opam ecosystem. I developed an reproducible testing opam repository with opam packages that do not depend on OCaml, mainly for further tooling development. Some tests were also conducted on a Debian system with the same result. The variations, apart from build time, were using a different user, and different locale settings.

As mentioned above, more environment, such as the CPU features, and external system packages, should be captured in the build environment.

When comparing OCaml libraries, some output files (cmt / cmti / cma / cmxa) are not deterministic, but contain minimal diverge where I was not able to spot the root cause. It would be great to fix this, likely in the OCaml compiler distribution. Since the final result, the binary I'm interested in, is not affected by non-identical intermediate build products, I hope someone (you?) is interested in improving on this side. OCaml bytecode output also seems to be non-deterministic. There is a discussion on the coq issue tracker which may be related.

In contrast to initial plans, I did not used the BUILD_PATH_PREFIX_MAP environment variable, which is implemented in OCaml by PR #1515 (and followups). The main reasons are that something in the OCaml toolchain (I suspect the bytecode interpreter) needed absolute paths to find libraries, thus I'd need a symlink from the left-hand side to the current build directory, which was tedious. Also, my installed assembler does not respect the build path prefix map, and BUILD_PATH_PREFIX_MAP is not widely supported. See e.g. the Debian zarith package with different build paths and its effects on the binary.

I'm fine with recording the build path (switch location) in the build environment for now - it turns out to end up only once in MirageOS unikernels, likely by the last linking step, which hopefully soon be solved by llvm 9.0.

What was fun was to compare the unikernel when built on Linux with gcc against a built on FreeBSD with clang and lld - spoiler: they emit debug sections with different dwarf versions, it is pretty big. Other fun differences were between OCaml compiler versions: the difference between minor versions (4.08.0 vs 4.08.1) is pretty small (~100kB as human-readable output), while the difference between major version (4.08.1 vs 4.09.0) is rather big (~900kB as human-readable diff).

An item on my list for the future is to distribute the opam export, build hashes and build environment artifacts in a authenticated way. I want to integrate this as in-toto style into conex, my not-yet-deployed implementation of tuf for opam that needs further development and a test installation, hopefully in 2020.

If you want to support our work on MirageOS unikernels, please donate to robur. I'm interested in feedback, either via twitter, hannesm@mastodon.social or via eMail.

X509 0.7

hannes — 2019-08-15T11:21:30-00:00

Cryptographic material

Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. X.509 is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is available here.

OCaml ecosystem evolving

More than 5 years ago David Kaloper and I released the initial ocaml-x509 package as part of our TLS stack, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in RFC 5280). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's asn1-combinators library changed much over time.

The OCaml ecosystem evolved over the years, which lead to some changes:

Camlp4 deprecation - we used camlp4 for stream parsers of PEM-encoded certificates, and sexplib.syntax to derive s-expression decoders and encoders; +
Avoiding brittle ppx converters - which we used for s-expression decoders and encoders of certificates after camlp4 was deprecated; +
Build and release system iterations - initially oasis and a packed library, then topkg and ocamlbuild, now dune; +
Introduction of the result type in the standard library - we used to use [ `Ok of certificate option | `Fail of failure ]; +
No more leaking exceptions in the public API; +
Usage of pretty-printers, esp with the fmt library val pp : Format.formatter -> 'a -> unit, instead of val to_string : t -> string functions; +
Release of ptime, a platform-independent POSIX time support; +
Release of rresult, which includes combinators for computation results; +
Release of gmap, a Map whose value types depend on the key, used for X.509 extensions, GeneralName, DistinguishedName, etc.; +
Release of domain-name, a library for domain name operations (as specified in RFC 1035) - used for name validation; +
Usage of the alcotest unit testing framework (instead of oUnit). +

More use cases for X.509

Initially, we designed and used ocaml-x509 for providing TLS server endpoints and validation in TLS clients - mostly on the public web, where each operating system ships a set of ~100 trust anchors to validate any web server certificate against. But once you have a X.509 implementation, every authentication problem can be solved by applying it.

Authentication with path building

It turns out that the trust anchor sets are not equal across operating systems and versions, thus some web servers serve sets, instead of chains, of certificates - as described in RFC 4158, where the client implementation needs to build valid paths and accept a connection if any path can be validated. The path building was initially in 0.5.2 slightly wrong, but fixed quickly in 0.5.3.

Fingerprint authentication

The chain of trust validation is useful for the open web, where you as software developer don't know to which remote endpoint your software will ever connect to - as long as the remote has a certificate signed (via intermediates) by any of the trust anchors. In the early days, before let's encrypt was launched and embedded as trust anchors (or cross-signed by already deployed trust anchors), operators needed to pay for a certificate - a business model where some CAs did not bother to check the authenticity of a certificate signing request, and thus random people owning valid certificates for microsoft.com or google.com.

Instead of using the set of trust anchors, the fingerprint of the server certificate, or preferably the fingerprint of the public key of the certificate, can be used for authentication, as optionally done since some years in jackline, an XMPP client. Support for this certificate / public key pinning was added in x509 0.2.1 / 0.5.0.

Certificate signing requests

Until x509 0.4.0 there was no support for generating certificate signing requests (CSR), as defined in PKCS 10, which are self-signed blobs containing a public key, an identity, and possibly extensions. Such as CSR is sent to the certificate authority, and after validation of ownership of the identity and paying a fee, the certificate is issued. Let's encrypt specified the ACME protocol which automates the proof of ownership: they provide a HTTP API for requesting a challenge, providing the response (the proof of ownership) via HTTP or DNS, and then allow the submission of a CSR and downloading the signed certificate. The ocaml-x509 library provides operations for creating such a CSR, and also for signing a CSR to generate a certificate.

Mindy developed the command-line utility certify which uses these operations from the ocaml-x509 library and acts as a swiss-army knife purely in OCaml for these required operations.

Maker developed a let's encrypt library which implements the above mentioned ACME protocol for provisioning CSR to certificates, also using our ocaml-x509 library.

To complete the required certificate authority functionality, in x509 0.6.0 certificate revocation lists, both validation and signing, was implemented.

Deploying unikernels

As described in another post, I developed albatross, an orchestration system for MirageOS unikernels. This uses ASN.1 for internal socket communication and allows remote management via a TLS connection which is mutually authenticated with a X.509 client certificate. To encrypt the X.509 client certificate, first a TLS handshake where the server authenticates itself to the client is established, and over that connection another TLS handshake is established where the client certificate is requested. Note that this mechanism can be dropped with TLS 1.3, since there the certificates are transmitted over an already encrypted channel.

The client certificate already contains the command to execute remotely - as a custom extension, being it "show me the console output", or "destroy the unikernel with name = YYY", or "deploy the included unikernel image". The advantage is that the commands are already authenticated, and there is no need for developing an ad-hoc protocol on top of the TLS session. The resource limits, assigned by the authority, are also part of the certificate chain - i.e. the number of unikernels, access to network bridges, available accumulated memory, accumulated size for block devices, are constrained by the certificate chain presented to the server, and currently running unikernels. The names of the chain are used for access control - if Alice and Bob have intermediate certificates from the same CA, neither Alice may manage Bob's unikernels, nor Bob may manage Alice's unikernels. I'm using albatross since 2.5 years in production on two physical machines with ~20 unikernels total (multiple users, multiple administrative domains), and it works stable and is much nicer to deal with than scp and custom hacked shell scripts.

Why 0.7?

There are still some missing pieces in our ocaml-x509 implementation, namely modern ECC certificates (depending on elliptic curve primitives not yet available in OCaml), RSA-PSS signing (should be straightforward), PKCS 12 (there is a pull request, but this should wait until asn1-combinators supports the ANY defined BY construct to cleanup the code), ... +Once these features are supported, the library should likely be named PKCS since it supports more than X.509, and released as 1.0.

The 0.7 release series moved a lot of modules and function names around, thus it is a major breaking release. By using a map instead of lists for extensions, GeneralName, ..., the API was further revised - invariants that each extension key (an ASN.1 object identifier) may occur at most once are now enforced. By not leaking exceptions through the public interface, the API is easier to use safely - see let's encrypt, openvpn, certify, tls, capnp, albatross.

I intended in 0.7.0 to have much more precise types, esp. for the SubjectAlternativeName (SAN) extension that uses a GeneralName, but it turns out the GeneralName is as well used for NameConstraints (NC) in a different way -- IP in SAN is an IPv4 or IPv6 address, in CN it is the IP/netmask; DNS is a domain name in SAN, in CN it is a name starting with a leading dot (i.e. ".example.com"), which is not a valid domain name. In 0.7.1, based on a bug report, I had to revert these variants and use less precise types.

Conclusion

The work on X.509 was sponsored by OCaml Labs. You can support our work at robur by a donation, which we will use to work on our OCaml and MirageOS projects. You can also reach out to us to realize commercial products.

I'm interested in feedback, either via ~~twitter~~ hannesm@mastodon.social or via eMail.

Building reproducible Python environments with XARs

janestreet — 2023-04-14T00:00:00-00:00

Our traders and researchers love Python for its agility and for its huge +open-source ecosystem, especially when it comes to machine learning. But +the heavy use of notebooks can make it difficult to support. Notebooks +have a very different lifecycle than regular code, and aren’t always +rigorously version controlled. And while most of our code (much of it +written in OCaml) lives in a monorepo, putting all notebooks there is +difficult; many notebooks end up being stored all over the place.

+ +

What if writing tests was a joyful experience?

janestreet — 2023-01-09T00:00:00-00:00

At Jane Street we use a pattern/library called “expect tests” that +makes test-writing feel like a REPL session, or like exploratory +programming in a Jupyter notebook—with feedback cycles so fast and +joyful that it feels almost tactile. Having used them for some time now +this is the only way I’d ever want to write tests.

+ +

Accelerating zk-SNARKs - MSM and NTT algorithms on FPGAs with Hardcaml

janestreet — 2022-12-07T00:00:00-00:00

In 2022 a consortium of companies ran an international competition, +called the ZPrize, to advance the state of +the art in “zero-knowledge” cryptography. We decided to have a go in +our free time at submitting solutions to both the Multi-Scalar +Multiplication (MSM) and Number Theoretic Transform (NTT) tracks, +using the same open source Hardcaml libraries +that Jane Street uses for our own FPGA development. We believe by +using Hardcaml we were able to more efficiently and robustly come up +with designs in the short competition period. These designs also +interact with the standard vendor RTL flow and so we hope they will be +useful to others.

+ +

Visualizing information propagation in markets

janestreet — 2022-11-23T00:00:00-00:00

The Dojima rice market, established around 1716, is widely considered to +be the world’s first organized futures exchange. Instead of directly +exchanging money for rice on the spot, merchants would agree on a price +and future date at which rice and money would be exchanged. This allowed +farmers and consumers to hedge their risk. As a result, information +about the abundance or lack of rice would travel across the country as +fast as rice merchants carried it.

+ +

Computations that differentiate, debug, and document themselves

janestreet — 2022-11-17T00:00:00-00:00

One of the problems we wrestle with at Jane Street is how to +understand and manage the costs associated with the positions we hold: +things like margin, financing costs, market risk, regulatory capital +requirements, and so on. To that end, we’ve built systems that +estimate these costs and propose ways to reduce them. Essentially, +this is a numerical optimization problem.

+ +

Introducing the Jane Street Graduate Research Fellowship

janestreet — 2022-08-30T00:00:00-00:00

We are excited to announce the launch of the Jane Street Graduate Research Fellowship!

+ +

What the interns have wrought, 2022 edition

janestreet — 2022-08-25T00:00:00-00:00

We’re once again at the end of our internship season, and it’s my task +to provide a few highlights of what the interns accomplished while +they were here.

+ +

Research internships in our Tools and Compilers group

janestreet — 2022-03-04T00:00:00-00:00

We are excited to announce research internships in our Tools and +Compilers group.

+ +

How Jane Street Pairs Interns to Projects and Teams During the Software Engineering Internship

janestreet — 2022-01-14T00:00:00-00:00

Software engineering intern candidates often ask how team placement +works and how much input incoming interns have over their teams and +projects. We know team placement is an important factor for many +students when deciding which internship to accept. We’ve spent +considerable time and thought on this process in recent years and hope +to demystify the experience with this post. ¹

+ +

+
This process is used in New York and London. Due to their smaller size Hong Kong’s process is slightly different ↩
+

Magic-trace: Diagnosing tricky performance issues easily with Intel Processor Trace

janestreet — 2022-01-11T00:00:00-00:00

Intel Processor Trace is a hardware technology that can record all +program execution flow along with timing information accurate to +around 30ns. As far as I can tell a l m o s t +nobody uses it, seemingly because capturing the data is tricky and, +without any visualization tools, you’re forced to read enormous text +dumps.

+ +

Hiring a Developer Educator

janestreet — 2021-10-21T00:00:00-00:00

We spend a lot of time on education at Jane Street. Like, really a +lot.

+ +

Goodbye Core_kernel

janestreet — 2021-08-26T00:00:00-00:00

We recently restructured our standard libraries at Jane Street in a +way that eliminates the difference between Core_kernel and Core +and we’re happy with the result. The new layout should reach the open +source world before the end of the year.

+ +

What the interns have wrought, 2021 edition

janestreet — 2021-08-09T00:00:00-00:00

It’s the end of another dev internship season, and this one marked +something of a transition, since halfway through the season, NY-based +interns were invited back to the recently reinvigorated office. Which +means that many more of us got the chance to meet and hang out with +the interns in person than we did last year. And hopefully the +interns were able to get a better sense of Jane Street and how it +operates.

+ +

Looking for a developer experience engineer

janestreet — 2021-06-15T00:00:00-00:00

This role has been filled

+ +

Growing the Hardcaml toolset

janestreet — 2020-12-01T00:00:00-00:00

I am pleased to announce that we have recently released a slew of new +Hardcaml libraries!

+ +

Announcing Our Market Prediction Kaggle Competition

janestreet — 2020-11-24T00:00:00-00:00

Jane Street is running a Kaggle contest based on a real problem with +real financial data. If you like ML projects, or think you might, +head over and check it +out. +We think it’s a pretty fun one. The prizes are pretty good too, with a +total $100K being paid out.

+ +

Finding memory leaks with Memtrace

janestreet — 2020-10-06T00:00:00-00:00

Memory issues can be hard to track down. A function that only +allocates a few small objects can cause a space leak if it’s called +often enough and those objects are never collected. Even then, many +objects are supposed to be long-lived. How can a tool, armed with data +on allocations and their lifetimes, +help sort out the expected from the suspicious?

+ +

Memory allocator showdown

janestreet — 2020-09-15T00:00:00-00:00

Since version 4.10, OCaml offers a new best-fit memory allocator +alongside its existing default, the next-fit allocator. At Jane +Street, we've seen a big improvement after switching over to the new +allocator. + +This post isn't about how the new allocator works. For that, the best +source is these notes from a talk by its +author. Instead, this post is about just how tricky it is to compare two +allocators in a reasonable way, especially for a garbage-collected +system. + +

Announcing Signals and Threads, a new podcast from Jane Street

janestreet — 2020-08-31T00:00:00-00:00

I’m excited (and slightly terrified) to announce that Jane Street is +releasing a new podcast, called Signals and +Threads, and I’m going to be the +host.

+ +

What the interns have wrought, 2020 edition

janestreet — 2020-08-17T00:00:00-00:00

It’s been an unusual internship season.

+ +

The Jane Street Interview Process — 2020 Edition

janestreet — 2020-07-24T00:00:00-00:00

We’re busy preparing for our software engineering fall hiring +season. Over the years we’ve +done our best to make our interview process more transparent to +candidates. While many candidates show up knowing something about what +our interviews look like, much of the information floating around on +the internet is outdated or wrong. These past few months have also +changed a lot about the process as we’ve adapted to working from home +and other effects of COVID-19.

+ +

Really low latency multipliers and cryptographic puzzles

janestreet — 2020-06-22T00:00:00-00:00

At Jane Street, we have some experience using FPGAs for low-latency +systems–FPGAs are programmable hardware where you get the speed of an +application-specific integrated circuit (ASIC) but without being +committed to a design that’s burned into the chip. It wasn’t so long +ago that FPGAs were expensive and rare, but these days, you can rent a +$5,000 card on the Amazon AWS cloud for less than $3 an hour.

+ +

Using ASCII waveforms to test hardware designs

janestreet — 2020-06-01T00:00:00-00:00

At Jane Street, an “expect +test” is a +test where you don’t manually write the output you’d like to check +your code against – instead, this output is captured automatically +and inserted by a tool into the testing code itself. If further runs +produce different output, the test fails, and you’re presented with +the diff.

+ +

Chrome extensions: Finding the missing proof

janestreet — 2020-04-17T00:00:00-00:00

Web browsers have supported custom +plug-ins and +extensions since +the 1990s, giving users the ability to add their own features and +tools for improving workflow or building closer integration with +applications or databases running on back-end servers.

+ +

Watch all of Jane Street's tech talks

janestreet — 2020-02-20T00:00:00-00:00

Jane Street has been posting tech talks from internal speakers and +invited guests for years—and they’re all available on our YouTube +channel:

+ +

Troubleshooting systemd with SystemTap

janestreet — 2020-02-03T00:00:00-00:00

When we set up a schedule on a computer, such as a list of commands to +run every day at particular times via Linux cron +jobs, we +expect that schedule to execute reliably. Of course we’ll check the +logs to see whether the job has failed, but we never question whether +the cron daemon itself will function. We always assume that it will, +as it always has done; we are not expecting mutiny in the ranks of the +operating system.

+ +

Using Python and OCaml in the same Jupyter notebook

janestreet — 2019-12-16T00:00:00-00:00

+The cover image is based on Jupiter family by NASA/JPL. +

+ +

Deep-Learning the Hardest Go Problem in the World

janestreet — 2019-12-06T00:00:00-00:00

Updates and a New Run

+ +

Commas in big numbers everywhere: An OpenType adventure

janestreet — 2019-10-14T00:00:00-00:00

My job involves a lot of staring at large numbers, mostly latencies in +nanoseconds, and picking out magnitudes like microseconds. I noticed +myself constantly counting digits in my text editor, in my terminal, +and in Jupyter notebooks in my browser.

+ +

What the interns have wrought, 2019 edition

janestreet — 2019-08-30T00:00:00-00:00

Jane Street’s intern program yet again is coming to an end, which is a +nice opportunity to look back over the summer and see what they’ve +accomplished.

+ +

Using OCaml to drive a Raspberry Pi robot car

janestreet — 2019-08-19T00:00:00-00:00

Back when the Raspberry Pi was first released in 2012 Michael Bacarella wrote +a blog post +on using OCaml and Async on this little device. +Since then installing OCaml via opam has become a pretty smooth experience +and everything works out of the box when using Raspbian – the default Raspberry Pi +distribution.

+ +

Do applied programming languages research at Jane Street!

janestreet — 2019-08-16T00:00:00-00:00

As our Tools & Compilers team has grown, the kinds of projects we work +on has become more ambitious. Here are some of the major things we’re +currently working on:

+ +

A look at OCaml 4.08

janestreet — 2019-07-12T00:00:00-00:00

Now that OCaml 4.08 has been released, let’s have a look at what was +accomplished, with a particular focus on how our plans for +4.08 fared. I’ll mostly focus on work that we +in the Jane Street Tools & Compilers team were involved with, but we are +just some of the contributors to the OCaml compiler, and I’ll have a +quick look at the end of the post at some of the other work that went +into 4.08.

+ +

Of Pythons and Camels

janestreet — 2019-07-09T00:00:00-00:00

Welcome to another post in our series of how to use OCaml for machine learning. +In previous posts we’ve discussed artistic style-transfer and +reinforcement learning. If you haven’t read these feel +free to do so now, we’ll wait right here until you’re done. Ready? Ok, let’s +continue …

+ +

Thoughts from AAAI 2019

janestreet — 2019-05-13T00:00:00-00:00

At Jane Street, for the last several years, we have been increasingly interested +in machine learning and its many use cases. This is why it was exciting when +earlier this year myself and a few of my colleagues had the opportunity to +attend the AAAI 2019 conference. We’d like to take this space to share with you +some of the interesting projects and themes we saw at the conference.

+ +

Learning ML Depth-First

janestreet — 2019-04-17T00:00:00-00:00

If you haven’t heard of it, Depth First +Learning is a +wonderful resource for learning about machine learning.

+ +

Machining the ultimate hackathon prize

janestreet — 2019-02-28T00:00:00-00:00

Jane Street is sponsoring this year’s MakeMIT +hackathon, and we wanted to create a prize for +the winners that would do justice to the maker spirit of the +competition. As makers ourselves – it’s not unusual to find a +“software” engineer here who hacks on FPGAs or who has a CNC machine +at home – it felt natural to get our hands dirty.

+ +

Accelerating Self-Play Learning in Go

janestreet — 2019-02-28T00:00:00-00:00

At Jane Street, over the last few years, we’ve been increasingly exploring machine learning to improve our models. Many of us are fascinated by the rapid improvement we see in a wide variety of applications due to developments in deep learning and reinforcement learning, both for its exciting potential for our own problems, and also on a personal level of pure interest and curiosity outside of work.

+ +

Playing Atari Games with OCaml and Deep Reinforcement Learning

janestreet — 2019-02-02T00:00:00-00:00

In a previous blog post +we detailed how we used OCaml to reproduce some classical deep-learning results +that would usually be implemented in Python. Here we will do the same with +some Reinforcement Learning (RL) experiments.

+ +

L2 Regularization and Batch Norm

janestreet — 2019-01-29T00:00:00-00:00

This blog post is about an interesting detail about machine learning +that I came across as a researcher at Jane Street - that of the +interaction between L2 regularization, also known as +weight decay, and batch normalization.

+ +

A tutorial for building web applications with Incr_dom

janestreet — 2019-01-15T00:00:00-00:00

At Jane Street, our web UIs are built on top of an in-house framework +called Incr_dom, modeled in +part on React’s virtual +DOM. Rendering different +views efficiently in response to changes made to a shared model is a +quintessentially incremental computation—so it should be no surprise +that Incr_dom is built on top of +Incremental.

+ +

How to shuffle a big dataset

janestreet — 2018-09-26T00:00:00-00:00

At Jane Street, we often work with data that has a very low +signal-to-noise ratio, but fortunately we also have a lot of data. +Where practitioners in many fields might be accustomed to +having tens or hundreds of thousands of correctly labeled +examples, some of our problems are more like having a billion training +examples whose labels have only a slight tendency to be correct. +These large datasets present a number of interesting engineering +challenges. The one we address here: How do you shuffle a really +large dataset? (If you’re not familiar with why one might need this, +jump to the section Why shuffle below.)

+ +

Deep learning experiments in OCaml

janestreet — 2018-09-20T00:00:00-00:00

Last year we held a machine learning seminar in our London office, +which was an opportunity to reproduce some classical deep learning +results with a nice twist: we used OCaml as a programming language +rather than Python. This allowed us to train models defined in a +functional way in OCaml on a GPU using TensorFlow.

+ +

What the interns have wrought, 2018 edition

janestreet — 2018-08-06T00:00:00-00:00

Yet again, intern season is coming to a close, and so it’s time to +look back at what the interns have achieved in their short time with +us. I’m always impressed by what our interns manage to squeeze into +the summer, and this year is no different.

+ +

Plans for OCaml 4.08

janestreet — 2018-06-29T00:00:00-00:00

With the external release of OCaml 4.07.0 imminent, we in Jane Street’s +Tools & Compilers group have been planning what we want to work on for +inclusion in OCaml 4.08. These days OCaml uses (or at least attempts) a +time-based release process with releases scheduled every 6 months. We’re +trying to avoid rushing in changes at the last minute – as we’ve been +prone to do in the past – so this list is restricted to things we could +conceivably finish in the next 4-5 months.

+ +

Repeatable exploratory programming

janestreet — 2018-04-22T00:00:00-00:00

Expect tests are a technique I’ve written about +before, but until recently, it’s been a +little on the theoretical side. That’s because it’s been hard to take +these ideas out for a spin due to lack of tooling outside of Jane +Street’s walls.

+ +

OCaml all the way down

janestreet — 2018-04-04T00:00:00-00:00

One of the joys of working at Jane Street for the last 15 or so years +has been seeing how our software stack has grown in scope. When I +started, I was building pretty narrowly focused systems for doing +statistical research on trading strategies, and then building systems +for executing those same strategies.

+ +

Putting the I back in IDE: Towards a Github Explorer

janestreet — 2018-03-27T00:00:00-00:00

Imagine a system for editing and reviewing code where:

+ +

Learn OCaml in NYC

janestreet — 2018-02-16T00:00:00-00:00

Interested in learning OCaml? In the NYC area? Then this might +be for you!

+ +

Proofs (and Refutations) using Z3

janestreet — 2018-02-15T00:00:00-00:00

People often think of formal methods and theorem provers as forbidding +tools, cool in theory but with a steep learning curve that makes them +hard to use in real life. In this post, we’re going to describe a case +we ran into recently where we were able to leverage theorem proving +technology, Z3 in particular, to validate some real world engineering +we were doing on the OCaml compiler. This post is aimed at readers +interested in compilers, but assumes no familiarity with actual +compiler development.

+ +

Work on the OCaml compiler at Jane Street!

janestreet — 2017-12-20T00:00:00-00:00

As Jane Street grows, the quality of the development tools we use +matters more and more. We increasingly work on the OCaml compiler +itself: adding useful language features, fine-tuning the type system +and improving the performance of the generated code. Alongside this, +we also work on the surrounding toolchain, developing new tools for +profiling, debugging, documentation and build automation.

+ +

Does batch size matter?

janestreet — 2017-10-31T00:00:00-00:00

This post is aimed at readers who are already familiar with +stochastic gradient descent +(SGD) and terms like “batch size”. For an introduction to these +ideas, I recommend Goodfellow et al.’s +Deep Learning, in particular the +introduction and, for more about SGD, Chapter 8. The relevance of SGD +is that it has made it feasible to work with much more complex models +than was formerly possible.

+ +

How Jane Street Does Code Review (Jane Street Tech Talk)

janestreet — 2017-10-29T00:00:00-00:00

It’s time for our next +Jane Street Tech Talk. When +we’ve solicited suggestions for topics, one common request has been to +talk about our internal development process. Our next talk, +How Jane Street Does Code Review, +should fit the bill. The talk is being given by our own Ian Henry, and +discusses how we approach code review, and in particular how Iron, the +code review system we’ve been using and improving for some years now, +fits in to that process.

+ +

Jane Street Tech Talk, Verifying Network Data Planes

janestreet — 2017-09-26T00:00:00-00:00

After a summer hiatus, the Jane Street Tech Talks series is back on +for the fall! Last we left it, our very own Dominick LoBraico +presented on the evolution of our internal configuration methodology +and the systems that support it. For anybody that missed it, you can +check out a recording of the talk on YouTube.

+ +

Real world machine learning (part 1)

janestreet — 2017-08-28T00:00:00-00:00

Trading is a competitive business. You need great people and great +technology, of course, but also trading strategies that make money. +Where do those strategies come from? In this post we’ll discuss how +the interplay of data, math and technology informs how we develop and +run strategies.

+ +

How to design a tree diffing algorithm

janestreet — 2017-08-25T00:00:00-00:00

For those of you interested in what +what +interns +do at Jane Street, here’s a +post from former intern +Tristan Hume, on his work developing tree-diffing algorithms last +summer at Jane Street. It’s a fun (and very detailed!) read.

Ironing out your development style

janestreet — 2017-08-24T00:00:00-00:00

People seem to enjoy talking about programming methodologies. They +give them cute names, like +eXtreme programming, +Agile, and +Scrum; run +conferences and build +communities around them; +write +books +that describe how to use them in excruciating detail; and +manifestos that lay out their +philosophy.

+ +

Hiring an FPGA engineer

janestreet — 2017-08-16T00:00:00-00:00

Jane Street is looking to hire an engineer with experience in both +software and hardware design to work on FPGA-based applications, and +on tools for creating such applications.

+ +

What the interns have wrought, 2017 edition

janestreet — 2017-08-14T00:00:00-00:00

Intern season is coming to a close, and it’s a nice time to look back +(as I’ve done in +previous +years) and review some of what +the interns did while they were here. The dev intern program has grown +considerably, with almost 40 dev interns between our NY, London, and +Hong Kong offices.

+ +

When Bash Scripts Bite

janestreet — 2017-05-11T00:00:00-00:00

There are abundant resources online trying to scare programmers away from using +shell scripts. Most of them, if anything, succeed in convincing the reader to +blindly put something that resembles

+ +

Looking for a technical writer

janestreet — 2017-05-01T00:00:00-00:00

Update: I’m excited to say that we’ve now hired a (great!) technical +writer, so the position is closed.

+ +

Caveat Configurator: how to replace configs with code, and why you might not want to

janestreet — 2017-04-25T00:00:00-00:00

We have a new tech talk coming up on +May 17th, from our very own Dominick LoBraico. This one is about how to +represent configurations with programs. In some sense, this is an obvious idea. +Lots of programmers have experienced the dysphoria that comes from watching your +elegant little configuration format metamorphize into a badly constructed +programming language with miserable tools. This happens because, as you try to +make your configs clearer and more concise, you often end up walking down the +primrose path of making your config format ever more language-like. But you +never really have the time to make it into a proper language.

+ +

This is not the performance you were looking for: the tricks systems play on us

janestreet — 2017-04-20T00:00:00-00:00

It’s often surprising just how much software performance depends on how the +software is deployed. All the time and effort you’ve invested in optimization +can be erased by a few bad decisions in scheduler policy, affinity, or +background workload on a server.

+ +

Trivial meta-programming with cinaps

janestreet — 2017-03-20T00:00:00-00:00

From now and then, I found myself having to write some mechanical and repetitive +code. The usual solution for this is to write a code generator; for instance in +the form of a ppx rewriter in the case of OCaml code. This however comes with a +cost: code generators are harder to review than plain code and it is a new +syntax to learn for other developers. So when the repetitive pattern is local to +a specific library or not widely used, it is often not worth the effort. +Especially if the code in question is meant to be reviewed and maintained by +several people.

+ +

One more talk, two more videos

janestreet — 2017-03-15T00:00:00-00:00

I’m happy to announce our next public tech +talk, called Seven +Implementations of Incremental, on Wednesday, April 5th, presented by yours +truly. You can register +here.

+ +

What a Jane Street software engineering interview is like

janestreet — 2017-02-28T00:00:00-00:00

Are you thinking about +applying to Jane Street +for a software engineering role? Or already have a phone interview scheduled but unsure +what to expect? Read on as we walk through an example phone interview with you.

+ +

Jane Street Tech Talks: Verifying Puppet Configs

janestreet — 2017-02-16T00:00:00-00:00

Our first Jane Street Tech Talk went really well! +Thanks to everyone who came and made it a fun event.

+ +

How to Build an Exchange

janestreet — 2017-01-11T00:00:00-00:00

UPDATE: We are full up. Tons of people signed up for the talk, and we’re +now at the limit of what we feel like we can support in the space. Thanks for +all the interest, and if you didn’t get into this one, don’t worry, we have more +talks coming!

+ +

A brief trip through Spacetime

janestreet — 2017-01-09T00:00:00-00:00

Spacetime is a new memory profiling facility for OCaml to help find space leaks +and unwanted allocations. Whilst still a little rough around the edges, we’ve +found it to be a very useful tool. Since there’s not much documentation for +using spacetime beyond this +readme, I’ve +written a little intro to give people an idea of how to use it.

+ +

A solution to the ppx versioning problem

janestreet — 2016-11-08T00:00:00-00:00

Ppx is a preprocessing system for OCaml where one maps over the OCaml abstract +syntax tree (AST) to interpret some special syntax fragments to generate code.

+ +

Observations of a functional programmer

janestreet — 2016-10-27T00:00:00-00:00

I was recently invited to do the keynote at the Commercial Users of Functional +Programming workshop, a 15-year-old gathering which is +attached to ICFP, the primary academic functional programming conference.

+ +

What the interns have wrought, 2016

janestreet — 2016-09-13T00:00:00-00:00

Now that the interns have mostly gone back to school, it’s a good time to look +back at what they did while they were here. We had a bumper crop – more than 30 +dev interns between our London, New York and Hong Kong offices – and they +worked on just about every corner of our code-base.

+ +

Unraveling of the tech hiring market

janestreet — 2016-08-31T00:00:00-00:00

Recruiting talented people has always been challenging.

+ +

Do you love dev tools? Come work at Jane Street.

janestreet — 2016-08-30T00:00:00-00:00

In the last few years, we’ve spent more and more effort working on developer +tools, to the point where we now have a tools-and-compilers group devoted to the +area, for which we’re actively hiring.

+ +

Let syntax, and why you should use it

janestreet — 2016-06-21T00:00:00-00:00

Earlier this year, we created +a ppx_let, a PPX rewriter that +introduces a syntax for working with monadic and applicative libraries like +Command, Async, Result and Incremental. We’ve now amassed about six months of +experience with it, and we’ve now seen enough to recommend it to a wider +audience.

+ +

ppx_core: context-free rewriters for better semantics and faster compilation

janestreet — 2016-05-23T00:00:00-00:00

At Jane Street, we have always been heavy users of pre-processors, first with +camlp4 and now ppx. Pre-processing makes the infrastructure a bit more complex, +but it save us a lot of time by taking care of a lot of tedious boilerplate code +and in some case makes the code a bit prettier.

+ +

Seven Implementations of Incremental

janestreet — 2016-03-09T00:00:00-00:00

We finally got a decent recording of one of my favorite talks. This one is about +our Incremental library (which I +wrote about here), and in particular about the +story of how we got to the present, quite performant, implementation.

+ +

OCaml 4.03: Everything else

janestreet — 2016-03-01T00:00:00-00:00

In my previous post I wrote about Flambda, which is the single +biggest feature coming to OCaml in this release. In this post, I’ll review the +other features of 4.03 that caught my eye.

+ +

A better inliner for OCaml, and why it matters

janestreet — 2016-02-24T00:00:00-00:00

OCaml 4.03 is branched and a first release candidate is imminent, so it seems +like a good time to take stock of what’s coming.

+ +

Self Adjusting DOM and Diffable Data

janestreet — 2016-02-10T00:00:00-00:00

In my last post, I gave some simple examples showing how +you could use +self adjusting computations, +or SAC, as embodied by our Incremental library, to +incrementalize the computation of virtual dom nodes. In this post, I’d like to +discuss how we can extend this approach to more realistic scales, and some of +the extensions to Incremental itself that are required to get there.

+ +

Self Adjusting DOM

janestreet — 2016-02-06T00:00:00-00:00

I’ve been thinking recently about how to +structure dynamic web applications, and in particular about the role that +incremental computation should play.

+ +

Incremental computation and the web

janestreet — 2016-01-30T00:00:00-00:00

I’ve recently been thinking about the world of JavaScript and web applications. +That’s odd for me, since I know almost nothing about the web. Indeed, Jane +Street’s use of web technologies is quite minimal – nearly all of our user +interfaces are text based, and all told we’ve been pretty happy with that.

+ +

Why OCaml?

janestreet — 2016-01-25T00:00:00-00:00

+ +

Testing with expectations

janestreet — 2015-12-02T00:00:00-00:00

Testing is important, and it’s hard to get people to do as much of it as they +should. Testing tools matter because the smoother the process is, the more tests +people will write.

+ +

Quickcheck for Core

janestreet — 2015-10-26T00:00:00-00:00

Automated testing is a powerful tool for finding bugs and specifying correctness +properties of code. Haskell’s Quickcheck library is the most well-known +automated testing library, based on over 15 years of research into how to write +property-base tests, generate useful sources of inputs, and report manageable +counterexamples. Jane Street’s Core library has not had anything comparable up +until now; version 113.00 of Core finally has a version of Quickcheck, +integrating automated testing with our other facilities like s-expression +reporting for counterexample values, and support for asynchronous tests using +Async.

+ +

rsync rounds timestamps to the nearest second

janestreet — 2015-10-07T00:00:00-00:00

I’m not sure how I’ve managed to use rsync for so many years without ever +noticing this, but hey, you learn something new every day!

+ +

No (functional) experience required

janestreet — 2015-08-19T00:00:00-00:00

Jane Street is a serious functional programming shop. We use OCaml, a statically +typed functional language for almost everything and have what is probably the +largest OCaml codebase anywhere.

+ +

Introducing Incremental

janestreet — 2015-07-18T00:00:00-00:00

I’m pleased to announce the release of +Incremental (well +commented mli +here), +a powerful library for building self-adjusting computations, i.e., +computations that can be updated efficiently when their inputs change.

+ +

Converting a code base from camlp4 to ppx

janestreet — 2015-07-08T00:00:00-00:00

As with many projects in the OCaml world, at Jane Street we have been working on +migrating from camlp4 to ppx. After having developed equivalent ppx rewriters +for our camlp4 syntax extensions, the last step is to actually translate the +code source of all our libraries and applications from the camlp4 syntax to the +standard OCaml syntax with extension points and attributes.

+ +

CPU Registers and OCaml

janestreet — 2015-05-05T00:00:00-00:00

Even though registers are a low-level CPU concept, having some knowledge about +them can help write faster code. Simply put, a CPU register is a storage for a +single variable. CPU can keep data in memory or cache or in registers and +registers are often much faster. Furthermore, some operations are possible only +when the data is in registers. Hence, the OCaml compiler tries to keep as many +variables as it can in the registers.

+ +

Reverse web proxy in ~50 lines of BASH

janestreet — 2015-05-01T00:00:00-00:00

In the spirit of reinventing the wheel for fun, I hacked this together as a +quick challenge to myself last week. It’s a little rough around the edges, but I +thought it was too cute not to share. If you have any bug fixes, please post +them in the comments.

+ +

Building a lower-latency GC

janestreet — 2015-04-10T00:00:00-00:00

We’ve been doing a bunch of work recently on improving the responsiveness of +OCaml’s garbage collector. I thought it would be worth discussing these +developments publicly to see if there was any useful feedback to be had on the +ideas that we’re investigating.

+ +

Faster OCaml to C calls

janestreet — 2015-04-09T00:00:00-00:00

The official OCaml documentation “Interfacing C with +OCaml” doesn’t +document some interesting performance features.

+ +

Why GADTs matter for performance

janestreet — 2015-03-30T00:00:00-00:00

When GADTs (Generalized Algebraic Data +Types) landed in +OCaml, I wasn’t particularly happy about it. I assumed that it was the kind of +nonsense you get when you let compiler writers design your programming language.

+ +

A lighter Core

janestreet — 2015-03-21T00:00:00-00:00

We recently released a version of our open source libraries with a much +anticipated +change +– Async_kernel, the heart of the Async concurrent programming library, now +depends only on Core_kernel rather than on Core.

+ +

Centralizing distributed version control, revisited

janestreet — 2015-03-04T00:00:00-00:00

7 years ago, I wrote a blog +post +about how we at Jane Street were using our distributed version control system +(hg, though the story would be the same for git) in a partially centralized +way. Essentially, we built a centralized repo and a continuous integration +system whose job was to merge in new changesets. The key responsibility of this +system was to make sure that a change was rejected unless it merged, compiled +and tested +cleanly.

+ +

Making making better

janestreet — 2015-01-31T00:00:00-00:00

We spend a lot of time and effort on training new people, and it never stops for +long. Right now our winter-intern class is ending; in five months we’ll have a +slew of new interns to get up to speed, and a few months after that we’ll have +an incoming class of new hires.

+ +

13 Virtues

janestreet — 2015-01-02T00:00:00-00:00

Very early on in his life, while on lengthy voyage from London to Philadelphia, +Ben Franklin created a system of thirteen virtues to live his life by. He spent +the remainder of his days giving special focus to one virtue per week in a 13 +week cycle, as well as noting the virtues he failed to live up to at the end of +each day.

+ +

Inspecting the Environment of a Running Process

janestreet — 2014-12-01T00:00:00-00:00

Sometimes its useful to be able see the values of environment variables in +running processes. We can use the following test program to see how well we can +accomplish this:

+ +

How to choose a teaching language

janestreet — 2014-11-17T00:00:00-00:00

If you were teaching a programming course, what language would you teach it in?

+ +

OCaml at MinidebConf TN 2023

tarides — 2023-04-28T00:00:00-00:00

Over the course of two days, I attended interesting sessions by speakers from across India and other parts of the world.

First Day

The conference was inaugurated by Dr. Ravikumar, a Member of Parliament (MP) from the Villupuram district. A tech-savvy politician who has presence in the fediverse, the MP emphasised the importance of adopting FOSS technologies by the government in his speech. He released the private beta of Prav, a privacy-focussed communication app.

The first session was "Introduction to Debian" by Sruthi Chandran, the first and only woman Debian Developer from India. It was interesting to see how the Debian community is comprised of a diverse set of people all across the world and is completely driven by volunteers. I learnt about the do-o-cratic model, where people doing the work make decisions.

A professor at RV College of Engineering and a FOSS enthusiast, Dr. Deepika's session on the KDE ecosystem was a great primer on motivating people to move to FOSS technologies. I found her suggestion to use the term Swatantra Software to indicate Free Software (Free as in Freedom) to be a great one. Then, Martha and Kelvin, mappers by profession, took us through the journey of OpenStreetMaps from a blank state to its growth of being at par with other maps. They did a quick session on how to contribute to it.

Later I presented a session on "Introducing a Code of Conduct" to an open-source community. This talk was built upon our experience of drafting and enforcing a Code of Conduct for the OCaml community, which led to completion in late 2022. This effort started earlier in the same year, with the idea of first forming a group of respected members in the community to act as the enforcement team. The effort was supported by the OCaml Software foundation. Once the team had enough strength, we worked on drafting a Code of Conduct document, largely inspired from existing texts, and iterating it over till it was accepted by the community.

The day ended with a speakers-only round table session of FOSSivist. It was a discussion with VGLUG volunteers on how to utilize Free and Open Source Software technologies to uplift the lives of underprivileged students in the district. For context, Villupuram falls in the bottom five in the literacy rate, both average and female literacy rate. VGLUG was formed by a group of volunteers actively working on identifying talented first-generation Villupuram learners and training them for a career in tech.

Second Day

The second day saw another lineup of interesting sessions. First was an introduction to contributing to Linux kernel by Nihal. This was followed by Gunnar Wolf's session on Debian authentication. It was evident Debian takes privacy seriously, and he urged the listeners to do so too. Bhuvana presented a session on why Diversity and Inclusion is important in tech.

This was followed by my presentation on An Invitation to OCaml. I talked about all the nice things OCaml offers, including the new and exciting features in OCaml 5 with Multicore support. The talk is aimed at folks who've been programming in other languages, but new to Functional Programming. We go over why FP, slowly moving on to talk about OCaml features like immutability, type inference, garbage collection, etc. We also briefly touch upon the new features in OCaml 5, namely native support for parallelism and concurrency. It was great to chat about functional programming with folks afterwards.

Renjith, an active Wikipedian, presented their story of moving a Malayalam daily newspaper called Janayugom to an entirely FOSS tech stack. This saved the company a lot of money and stopped Janayugom from shutting down. Renjith emphasised the importance of free speech in a democracy and how small maganizes and newspapers play a role in it. Then Subin presented Varnam, an Indic input tool.

I was impressed by the efforts taken by VGLUG volunteers and the Debian India team to organise everything in a smooth manner. From the time we landed in Villupuram, we did not worry about anything. Transport, food, and lodging were all taken care of by VGLUG volunteers. I did not think such a vibrant community of FOSS users would operate in a rural town. The community is doing great work to uplift the lives of people in their Villupuram.

Best of all, it was great to meet old friends and make new ones. I hope to spread the joy of OCaml in more places.

Compiler Hacking in Cambridge is Back!

tarides — 2023-03-22T00:00:00-00:00

What’s the best way to spend a Friday evening? We think most people would agree that hacking on OCaml is pretty much at the top of that list (although full disclosure, our sample size for this data could be larger).

On Friday the 24th of February, Tarides’s UK office hosted an evening of compiler hacking, presentations, and talks about all things OCaml. We’re continuing a tradition that began in 2013, making this our 19th event, when we (then known as OCaml Labs) were based at the Computer Lab in Cambridge. Just like back then, anyone with an interest in the OCaml compiler is welcome. At our recent event we had a mixture of students, industry professionals, and experts in attendance. If you'd like to create your own compiler hacking sessions, check out the wiki here.

Something that’s changed since 2013 is that OCaml now represents a large chunk of the undergraduate Computer Science tripos at the University of Cambridge; not only as the implementation language for courses such as Compiler Construction & Semantics of Programming Language, but literally as the first language students are taught! This means that we had quite a few undergraduates turn up – it was great to see such an interest in OCaml across different backgrounds.

+ + + + +

A Welcome and Introduction to the Compiler

The afternoon began with our very own David Allsopp giving the first of the day’s two talks. He briefly laid the foundation for what Tarides is and what we do, but focussed on introducing OCaml and outlining some examples of things to hack on. Since we had the pleasure of hosting many undergraduate students who were new to the OCaml community, as well as some grizzled veterans (sorry, Jon!), it was important to have a selection of projects for all abilities.

Suggestions included bug fixes (which are always welcomed), documentation edits and improvements (which are always needed), and issues labelled with the tag “good first issue” or “newcomer job.” Compilers that are self-bootstrapped (like OCaml) always require a complex build system, so David concluded with a demonstration of the sequence of build system targets, explaining each step along the way.

Once the introduction was over, the room settled into a hive of activity, with some people furiously typing and others scratching their heads and looking thoughtful. Many of the undergrads focused on getting familiar with the OCaml compiler, whereas more experienced developers began undertaking their own hacking projects. We had invited well-known OCaml compiler hackers (including some of our own) as an awesome resource for all levels of experience. A combination of in-person hacking and an informal setting provided the perfect environment for sharing imaginative new ideas - something we’ve all been missing since the pandemic.

With everyone divided up into smaller working groups, we worked our way around trying to help everyone make some progress. Groups were working on projects at all levels: some were trying to get the compiler to run hello world, whilst others (Patrick!) were forward-porting advanced modal type features between major versions of the compiler. A third year undergraduate was working on debugging the OCaml compiler for her dissertation, and she was attempting to use hash consing to make multiple identical values use the same bit of memory rather than multiple memory slots as a space-saving solution for the compiler.

+ + + + +

Local Allocations and Pizza

After a couple of hours of hacking, Stephen Dolan gave us a tour of his and Leo White’s ground-breaking work on stack allocation. This work was presented at ICFP 2022 and is a compiler feature that aims to improve performance by reducing heap allocations in OCaml programs. Local allocations let programs use space on the stack (instead of allocating on the heap), which is automatically reclaimed without requiring the assistance of the (resource-heavy) garbage collector (GC). OCaml uses a stop-the-world parallel garbage collector for collecting recently allocated objects in the minor heap. This means that all the OCaml threads will need to stop when the minor heap is collected. Generating fewer heap allocations means less garbage, and less garbage means improved performance and reduced pause times. This is particularly important for parallel workloads. Local allocations are already being run in production internally at Jane Street, and there are plans to bring the associated benefits to the masses by upstreaming the work to mainline OCaml.

After Stephen’s talk, and a quick but much needed pizza break, everyone went back to hacking. An all-too-common problem that cropped up several times happened when trying to run the freshly-built OCaml compiler from the build tree without first installing it. The error messages in this circumstance are not particularly intuitive, complaining of a "bad interpreter: no such file or directory." The message refers to a bootstrap issue; the program is trying to find the interpreter, but the interpreter hasn’t been installed yet. Some people solved the issue and moved straight onto the next task (a very common thing to do), but one group decided to tackle this head-on by improving the error message to provide more detail. This will help other new OCaml compiler developers and will almost certainly make life easier in our future hack events! This kind of “simple” fix is incredibly important for reducing the barrier to entry for new developers and emphasises the benefits of mixed-experience hack events with newcomers providing feedback and highlighting useful areas of improvement. We hope this work turns into a PR soon!

Another project focussed on ensuring OCaml programs can take advantage of new security features in Linux. There is a relatively new feature of the kernel that allows the user to create a secure temporary file that is isolated from other users. One participant was experimenting with different versions of OCaml and Linux to see how this feature might be used in OCaml. Implementing this in the Unix module is tempting, but as it provides the "lowest common denominator" interface, it has to be compatible with all platforms, and therefore does not cater to a niche function. A better option would be to write a separate library with a separate binding to address the compatibility issues, but that would require a lot of work for one feature. This illustrates the important kinds of questions that form the debate around supporting new, platform-specific features.

The prize for “oldest bug addressed” for the evening went to one of our most junior attendees, a first-year computer scientist who took on a problem first reported in 2005. The almost-20-year-old-issue involves structural comparisons of cyclical data structures and is easily reproduced by pasting “let rec x = 1 :: x in x = x” into a toplevel. A pull request fixing the problem was made during the evening and has generated a lot of interesting discussion!

Until Next Time

We’re thrilled that we could restart these events, and it was lovely to see so many familiar faces alongside all the newcomers. The next hack day is scheduled for March 31st, and we’re excited to see more people working on the compiler.

We’d love to see you at a future event, but even if you can’t come in person, there are loads of ways you can contribute. You can suggest projects and "good first issues," add and improve on documentation, and even set up your own local event! You can check out the wiki here.

We look forward to hanging out with more people around Cambridge who are curious or passionate about OCaml. If you’re interested in joining future events in Cambridge, please email us, we look forward to hearing from you! See you next time!

+ + + + + + + + + + +

More Than a Day: How Does Tarides Promote Women in Tech?

tarides — 2023-03-08T00:00:00-00:00

The aim of International Women’s Day is to raise awareness of gender inequalities and call for the empowerment of women worldwide. The goal is to forge a gender equal world and advance women’s equality in all forms.

Within the field of technology, the focus is on elevating and advancing gender parity in technology and celebrating the women shaping innovation.

While there have been advancements towards these goals, continuous attention and effort is required to create lasting change.

How is Tarides Promoting Gender Equality?

Externally

One of our goals is to foster diversity and inclusion in tech and help provide more opportunities for underrepresented groups, including women. As a company, we partner with and support several organisations that promote diversity in the field of computer science. Some of them include:

50 in Tech is working to achieve a gender balance of 50% women in tech by 2050. Their Gender Score Board helps companies across Europe measure their level of gender inclusion. They selected Tarides as an inclusive company and featured us on their website. For more information on our work with 50 in Tech, we have a blog post from 2022.
The Ada Tech School, named after the first computer programmer Ada Lovelace, is a programming school designed for women but open to all. They are driven by three values: feminism, empathy, and singularity. We have a blog post on our partnership with them from 2021.
GirlsCanCode is an initiative launched by the organisation Prologin that hosts summer camps specially aimed at teaching young women about computer programming, free of charge. We have a blog post about why we sponsor GirlsCanCode from 2022.
The Recurse Center is an initiative that offers educational retreats for anyone who wants to get better at programming. They also provide needs-based grants to traditionally underrepresented groups to make programming more accessible for all. We're happy to have hired many talented engineers from the Recurse Center over the years!
Outreachy is an internship program that provides paid remote internships in open source and open science. Outreachy’s goal is to increase diversity in open source and expressly invites anyone who faces underrepresentation or systemic bias in the technology industry of their country to apply. We sponsor and mentor interns in each biannual intake, and you can watch project presentations here (December 2021, May 2022, December 2022); read a community post about how to get involved as a mentor; and read a blog post from one of our Summer 2022 interns about her project.
The Oxbridge Women in Computer Science Conference is an annual one-day event hosted by the Universities of Oxford and Cambridge (UK). The purpose of the conference is to spotlight the successes of women within computer science and strengthen the network of women in computer science within a supportive environment. The conference is free and open to all genders. We have a blog post from 2020 on our sponsorship of the conference.

Internally

As a company we aim to provide a flexible and supportive working environment that encourages women to enter and remain in the workforce. Our aim is to make working at Tarides as inclusive as possible.

Examples of our policies include:

Childcare support as an employee perk
Flexible hours and working
Equal pay scales based on experience and skills
Apprenticeships and internships to kickstart careers, or to enable later-stage career changes
Career progression development in technical and managerial roles

Still Some Way to Go

There is still much room for improvement, and we continue to be committed to removing barriers for women: at Tarides, in open source, and in computer science.

Currently at Tarides, 24% of our workforce is female, with 15% in technical roles. Over the last 12 months, Tarides has grown from 55 to 83 people, with 27% of those hired being women. Despite this increase, we are still below our ambitious goal of reaching 30% of women in tech roles. This highlights the disappointing reality that for each position we want to hire for, there are still proportionally fewer women applying and reaching the later stages of recruitment in our field.

In tech, we can address the issues from a number of angles, all of which will improve the overall picture. Collectively, we still need to encourage girls into STEM areas at an early age, in order to gradually increase the numbers of women in tech overall, but also to increase the size of the potential hiring pool of female applicants. Having female role models is essential, and we must continue to increase the representation of women in technology and STEM fields to encourage girls, and women, to see themselves in these kinds of roles. We must also continue to support lifelong learning by funding and creating training opportunities and resources for later-stage career changes.

At Tarides, we have specifically noticed a skills and training gap between entry-level internships (e.g., Outreachy) and the next level of progression into junior software engineer. We are getting better at helping women make their first steps into the tech world, but where do they go next? This year, we are focussing on how we can specifically improve this by preparing more resources to help learn functional programming, OCaml, and open-source methods, and by understanding the different levels of training and education needed in order to progress beyond these initial stages.

Finally, gender equality is not just a topic we should consider on March 8th every year. We must ensure that equality is in everyone’s consciousness and that it forms the basis of our conversations and decisions.

+
“We will always have STEM with us. Some things will drop out of the public eye and will go away, but there will always be science, engineering and technology. And there will always, always be mathematics. Everything is physics and math.” - Katherine Johnson, NASA mathematician
+

The Journey to OCaml Multicore: Bringing Big Ideas to Life

tarides — 2023-03-02T00:00:00-00:00

Continuing our blog series on Multicore OCaml, this blog provides an overview of the road to OCaml Multicore. If you want to know how you can use OCaml 5 in your own projects, please contact us for more information. We also recommend watching KC Sivaramakrishnan's ICFP 22' talk Retrofitting Concurrency - Lessons from the Engine Room

The journey to Multicore OCaml is a journey from cutting-edge theory to real-life code. It’s the story of an idea that grew from a small side-project into a multinational effort that brought a long-awaited update to OCaml. Along the road, the Multicore OCaml team faced many different challenges, leading them to re-evaluate their priorities and approach tasks differently.

As part of the Multicore Project since December 2014, KC Sivaramakrishnan is in a good position to describe the process from the initial days of experimentation right up until launch. He has unique insight into the decisions, challenges, and successes that the team experienced as they worked to turn innovative ideas into tangible results.

The Journey Begins

In 2013, the world had survived the 21st of December 2012, Flappy Bird was popular, and everyone was doing the Harlem shake. At the University of Cambridge, Professor Anil Madhavapeddy launched the Multicore OCaml project as part of the OCaml Labs initiative alongside Leo White, Jeremy Yallop, and Phillipe Wang. They were eventually joined by Stephen Dolan, the then PhD student working on combining ML-style parameteric polymorphism with subtyping.

In 2014 KC, who had just finished his PhD in the US, joined the team. His PhD had focused on making a multicore version of MLton Standard ML compiler, which made him an asset to the growing team that would see the Multicore OCaml Project through to completion. Together they collaborated on a project that would see many partial victories and setbacks, before ultimately releasing OCaml 5.0 to the public in December 2022.

Timeline

In the years since the project started, there have been several developments and incremental successes. Below is an overview of the milestones along the road to Multicore OCaml:

+ + + + +

2013

Multicore OCaml project was started by Prof Anil Madhavapeddy in the OCaml Labs initiative at the University of Cambridge Computer Lab with Leo White, Jeremy Yallop, and Phillipe Wang. The team was later joined by the then PhD student Stephen Dolan, who was working on combining ML-style parameteric polymorphism with subtyping.

2014

March: Stephen Dolan, Leo White, and Anil Madhavapeddy started hacking on Multicore OCaml
March: Earliest commit that can be directly attributed to Multicore OCaml that is in the OCaml commit history. The commit removes most of the out-of-heap pointers the interpreter uses by replacing them with stack offsets.
September: A status update on Multicore OCaml was presented in the OCaml workshop 2014, you can read the associated paper by Stephen Dolan, Leo White, and Anil Madhavapeddy.

2015

First 6 months: An initial implementation of effect handlers was completed. The inspiration behind this idea came from the Eff language.
September: Effect handlers in OCaml were presented at the OCaml workshop 2015. You can read more about it in this blog post.

2016

May: Dagstuhl Seminar 16112: “From Theory to Practice of Algebraic Effects and Handlers.” Effect handlers in OCaml was presented and refined based on expert interactions.
ML workshop: Daniel Hilleström, Sam Lindley, KC Sivaramakrishnan "Compiling Links Effect Handlers to the OCaml Backend". Daniel Hillerstrom developed a Multicore OCaml backend for Links language compiling Links effect handlers to OCaml effect handlers.
ML workshop: Oleg Kiselyov and KC Sivaramakrishnan "Eff Directly in OCaml". Showed how to get the expressive power of Eff language directly using features from the OCaml language + OCaml effect handlers.
OCaml workshop: KC Sivaramakrishnan and Théo Laurent "Lock-Free Programming for the Masses". Presented the implementation of Reagents in OCaml, a composable lock-free programming library.

2017

Papers published at the ML & OCaml Workshop: Stephen Dolan, Spiros Eliopoulos, Daniel Hillerström, Anil Madhavapeddy, KC Sivaramakrishnan, and Leo White "Effectively Tackling the Awkward Squad". The work outlined in this paper showed how effect handlers can simplify concurrent systems programming. These ideas were then incorporated in the development of Eio.
Stephen Dolan and KC Sivaramakrishnan - "A Memory Model for Multicore OCaml". The paper proposed a relaxed memory model for OCaml, broadly following the design of axiomatic memory models for languages such as C++ and Java, but with a number of differences to provide stronger guarantees and easier reasoning to the programmer, at the expense of not admitting every possible optimisation. This work eventually lead to the relaxed memory model used in OCaml 5.

2018

Stephen Dolan, KC Sivaramakrishnan, Spiros Eliopoulos, Daniel Hillerström, Anil Madhavapeddy, and Leo White presented a forward looking paper on "Concurrent Systems Programming with Effect Handlers" at the Trends in Functional Programming conference. This is the full version of the 2017 ML Workshop paper.
Stephen Dolan, KC Sivaramakrishnan, and Anil Madhavapeddy, published a paper on the relaxed memory model for OCaml at PLDI, "Bounding Data Races in Space and Time". This is the full version of the memory model work presented at the 2017 OCaml Workshop.
The team worked on simplifying and speeding up the implementation of effect handlers.

2019

Sadiq Jaffer and Tom Kelly implemented a new garbage collector for the minor heap (parallel stop-the-world minor collector), which ensures that programs using C FFI in OCaml remain backwards compatible.
The Sandmark benchmark suite for rigorously benchmarking OCaml programs was developed and deployed. These days the performance of OCaml compiler is tracked continuously using the Sandmark nightly continuous benchmarking service.

2020

The team decided to switch to the parallel stop-the-world minor collector (ParMinor) as default and drop the support for the concurrent minor collector (ConcMinor). ParMinor GC avoided a breaking change in the C FFI introduced by the ConcMinor GC. One concern is that the stop-the-world aspect in ParMinor would be a scalability bottleneck at large core counts. Our performance evaluation on the Sandmark suite showed that the impact of ParMinor is minimal even at large core counts (120+).
KC Sivaramakrishnan, Stephen Dolan, Leo White, Sadiq Jaffer, Tom Kelly, Anmol Sahoo, Sudha Parimala, Atul Dhiman, and Anil Madhavapeddy presented "Retrofitting Parallelism onto OCaml" at ICFP 2020. The paper describes the design choices for multicore support in OCaml, the design of the ConcMinor and ParMinor GCs, detailed performance evaluation, and justifies our choice to switch to ParMinor as default. It won the distinguished paper award at ICFP.
From 2020 through 2021, the team focused on achieving feature parity with sequential OCaml (systhreads, GC performance, DWARF support, opam health check, etc.)

2021

KC Sivaramakrishnan, Stephen Dolan, Leo White, Sadiq Jaffer, Tom Kelly, and Anil Madhavapeddy published "Retrofitting Effect Handlers onto OCaml" at PLDI 2021. The paper describes the design choices for the concurrency substrate in OCaml 5 and how effect handlers are a good fit for our needs.
Later half of 2021, OCaml core developers began reviewing code for Multicore OCaml, including the new concurrency and parallelism features, see this document for more information.

2022

Early 2022, the Multicore PR was merged!
Significant efforts were made by core OCaml developers to implement new features, review them, and ready the compiler for release. Without their hard work and dedication, the would be no OCaml Multicore nor OCaml 5.0.
Memory model successfully implemented
RISC-V backend and ARM64 backend achieved
December 16th, 2022: OCaml 5.0 is released!

Why Multicore OCaml?

The number of cores on the machines that we use have been steadily increasing for years. Almost every computer now has several cores available to the user, and for a programming language to use them effectively it must support shared-memory parallel programming. If it does not, the user is forced to execute everything sequentially using only one core, or use multi-process programming, which is hard to use and in many cases less efficient than shared-memory parallel programming.

There are two main features coming with OCaml 5: Parallelism and Concurrency. Parallelism is about performance; it’s the idea that if you have an n amount of cores, you can make your program go faster by n amounts of time. The effects of parallelism will be most keenly felt in how fast your programs run, giving you as a user a significant performance boost.

On a bigger scale, parallel programming is significant for projects that need to complete resource intensive tasks quickly, like theorem provers for example. With multicore support for OCaml, developers can take advantage of features like type and memory-safety with unprecedented levels of performance.

Concurrency, on the other hand, is a programming abstraction. It is a way to tell your program that you want to execute several functions, each of which may potentially block for a short time while waiting for some external event. The programming language may choose either to execute such functions sequentially, one after the other, on a single core, interleaving their execution when a function gets blocked, or choose to execute them in parallel on several cores at once. Concurrency is useful, for example, when writing a web server that must handle several concurrent requests. The program may handle several such requests at the same time, but not necessarily need to use multiple cores to handle them. With OCaml 5, writing concurrent code is made a lot easier.

Previously, concurrent OCaml code would have to be written in a specific tool, Async or Lwt, that the developer would have to learn separately. However, these tools don’t currently allow for asynchronous and synchronous code to interact with each other. In a blog post from 2015,, Bob Nystrom goes describes this process in what he calls the ‘Functional Colouring Problem’. OCaml 5 brings in support for concurrency through effect handlers and the new Input/Output library Eio, which lets developers compose asynchronous and synchronous code together. It’s also easy to learn and use since it behaves like normal OCaml code, simplifying the developer workflow.

For those who still prefer to use Lwt or Async, OCaml 5 doesn’t preclude them from doing so. Should they one day want to switch from using either tool to using Eio, changing their code to be compatible with Eio is simple and user-friendly. Whilst they will still need to rewrite their applications to use the primitives provided by Eio, to do so is straight-forward and can be made incremental thanks to the Lwt- and Async-Eio bridges.

The team of people who worked on OCaml 5 knew from the start that bringing multicore support to OCaml would improve the lives of its users. It would make programs that run in OCaml faster and more efficient, as well as help developers be more productive.

The Academic and the Engineer

When academics are on the cutting edge of science, they're essentially creating a new area of research as they go. This leads to a natural lag time between innovation and the creation of academic papers revieweing the process. For example, KC explains that “The first talk on effect handlers was in 2015, but the first proper paper on effect handlers was just published in 2021.”

Refererring to the time between experimentation and finished product, KC goes on to say: “Personally, it has been challenging to take part in building these systems, because 95% of the work is not very visible but you have to create that 95% in order to talk about the 5%.”

Since the road to get here has been so long, it feels all the more exhilarating that release day has finally arrived.

+
“It’s incredible that we are at the stage where we’re able to take cutting-edge research and put it into practice. In the last few years, we’ve expanded from academic research to producing robust code that can be upstreamed.”
+

This is great news for the whole community, as it demonstrates OCaml’s potential to turn research into real products. The team behind the multicore effort’s goals were to modernise OCaml and make it faster and more efficient for everyone. The realisation of that goal took years of experimentation, optimisation, and groundbreaking research.

First Major Challenge: It’s All About Garbage

The first major challenge facing the Multicore team was OCaml’s garbage collector. In OCaml, there are two programs working together on the heap, the language and the garbage collector. If the language supports parallelism but the garbage collector does not, the language would run fast just to be slowed down by the garbage collector.

To avoid this problem, the team made the garbage collector support parallelism to give users a uniformly smooth experience. “Garbage collectors balance memory usage at the cost of time, so you can either have it use a small amount of memory but take a long time, or be fast but use a lot of memory,” KC comments.

With different variables to optimise for, the team had to make some crucial decisions. OCaml already had a user base with certain expectations. They had to ensure that their changes did not remove features that users had come to expect. For example, OCaml is a robust and predictable language, and they needed to replace the garbage collector without sacrificing on that predictability. They also didn’t want to settle for worse results in terms of performance.

Working on the new garbage collector, the team built an initial version that performed very well. However, they soon discovered that in order for the garbage collector to work, it would break the existing Application Programming Interface (API) interacting with C code.

+
“That was our dilemma: we had a nice, fast, garbage collector, but it would break people’s code.”
+

A broken API would have been bad news for anyone, but it would especially affect any existing projects that relied heavily on C code (like Coq), as well as many industrial users who would have had to change millions of lines of code. The team worried that this would create a fork in the community, between those who would find it worth the upgrade and those who would not.

This was a big lesson for the team: user friendliness is incredibly important when introducing new technologies, and a big part of user friendliness is backwards compatibility. With this in mind, they set out to redesign the garbage collector. Although they were initially resigned to sacrifice some performance for the sake of compatibility, they ended up with a final product that not only did not break any code, but also didn’t see significant performance losses! They presented their findings at ICFP 2020 and won the distinguished paper award.

Second Major Challenge: Memory Model, What Memory Model?

The second challenge came as a result of the very way computers are constructed. Unsurprisingly, the hardware that actually executes your code predates the multicore era. Consequently, the hardware and compilers running the code are designed to make optimisations based on the assumption that you’re running a single-threaded (so not multicore) program.

As you might imagine, several of these optimisations conflict with more modern, multicore aspects of code. In order for multicore code to run successfully in the face of these optimisations, useful abstractions are needed to determine what is safe and how parallel code is expected to run. These abstractions are called memory models, and they are necessary for hardware made for single-threaded programs to run multi-threaded code.

Memory models are very complex and have to balance simplicity with performance. The more straight-forward the model, the greater the risk that it can’t account for all possibilities, and therefore cause bugs. Conversely, if the memory model is complex enough to maximise performance, it will be hard for people to understand and use.

For the Multicore OCaml Project, the team decided to take inspiration from the memory models of C++ and Java, which choose to prioritise performance. However, they still wanted to make a memory model that was straightforward and intuitive. “OCaml is used to prove other languages, and if the memory model is too complicated, it becomes hard to verify other parallel programs,” KC explains.

By sacrificing a small amount of performance (around 3%), the team managed to create an OCaml memory model that was both high-performing and easy-to-use. The paper detailing the process is called Bounding Data Races in Space and Time.

In two of the big technological challenges that faced the team, a clear focus on user experience emerged. As a language with deep roots in academia, at times rumoured to be ‘difficult,’ focusing on improving user experience is an important part of making OCaml a language for everyone.

The People Behind the Project

Behind every project is a group of hardworking people. Stephen Dolan, Leo White, and Anil Madhavapeddy started the Multicore project back in 2014. Until 2018, KC, Stephen, and Leo were doing most of the hacking. After 2018, the team saw enormous growth with Sadiq Jaffer and Tom Kelly working together on the garbage collector. Today, there are around ten people hacking on Multicore OCaml at any given time, all working hard to ensure that OCaml 5 is a success.

The open-source community has also provided continuous, valuable feedback as work on Multicore OCaml has progressed. Every person who participates by sharing their opinions and experience helps the project more forward. Many core OCaml developers worked tirelessly to get OCaml 5.0 release ready. In particular we should highlight the support of Xavier Leroy, who spent a considerable amount of time and effort implementing changes to important pieces in the runtime to make them multicore compatible (such as closure representation, bytecode interpreter, etc.), as well as Gabriel Scherer for his enthusiastic support of Multicore features and the willingness to do an enormous amount of crucial work like reviewing a large number of Multicore PRs and additional features. The academic community has also actively utilised Multicore OCaml to push the boundaries of what is possible with effect handlers, and provided useful feedback and bug reports.

On the commercial side, Tezos has significantly helped the team test OCaml 5 by using multicore features for their tools PLONK prover. They’ve made good progress using OCaml 5 and have been extremely helpful by reporting on bugs and their experience.

Sandmark

Over the course of OCaml Multicore’s implementation, new tools have been developed to facilitate its creation. These tools are useful in and of themselves, and can be used in other projects. One tool born out of the OCaml Multicore push is the benchmarking suite Sandmark.

When Sadiq and Tom were working on the garbage collector, they had to understand how the change to a parallel garbage collector would affect non-parallel, sequential programs. To this end, they created Sandmark to benchmark different iterations.

Where Do We Go from Here?

OCaml 5 is just the beginning, and from its release springs countless more opportunities. Several teams across the community are innovating on new features for OCaml. These features are at various levels of maturity and development, with small groups of developers testing some of them, whilst others are more or less in the ideation phase. Some of these features in development are listed below, but this is by no means an exhaustive list:

Effects system: At the moment, there is no support from the OCaml type system to ensure that effect handlers are handled properly. An effect system is an extension of the type system that keeps track of which effects can be performed by an expression or a function, ensuring that effects are only performed in a context where a corresponding effect handler is set up to deal with them. In a language as well-established and large as OCaml, implementing new features comes with significant considerations. Backwards compatibility is a must, and the new system must work with the polymorphism, modularity, and generativity features already in place. For an early exploration of typed effect handlers in OCaml, check out Leo White's talk from 2018.
JavaScript: OCaml has a very nice compiler to JavaScript, but it couldn't compile effect handlers to JavaScript. Indeed, JavaScript does not provide a corresponding feature. A standard way to translate effect handlers is to transform the code into the so-called continuation-passing style (CPS). Functions require an extra argument: a one-argument continuation function. Instead of returning a result value, they call the continuation with this value. By making continuations explicit, one can then explicitly manipulate the control flow of the program, which makes it possible to support effect handlers. Js_of_ocaml has been recently modified to support effect handlers using this approach. This preliminary implementation has been released in Js_of_ocaml 5.0. It has provided their team with a good understanding of how effect handlers work and what technical difficulties exist when supporting them in a compiler targeting JavaScript. However, CPS transformation comes with an important negative impact on performance. The team then implemented a partial CPS transform that removed some of the overheads with the CPS transformation. There are still some overheads due to CPS conversion that can be eliminated with smarter analysis and transformation. The team is considering trying alternative compilation techniques to support effect handlers. For example, there are implementation strategies that should have low overhead as long as no effect is performed at the cost of making effect handling slower. However, this might make the generated code much larger. The CPS-based implementation provides them with a point of comparison for undertaking this work.
Local Allocations: implemented by Stephen Dolan and Leo White: This feature adds support for stack-allocated blocks. It enforces memory safety by requiring that heap-allocated blocks never point to stack-allocated blocks, and stack-allocated blocks never point to shorter-lived stack-allocated blocks. This is a big addition to OCaml’s type system and is still under development.
Unboxed Types: Currently in OCaml, all fields of a structure store values in a single-machine word. This word is further restricted by having to either point to a garbage-collected memory or be tagged to denote that the garbage collector should skip it. Unboxed types relax this restriction, allowing a field to hold values smaller or larger than a word. This can be used to save memory and to improve performances by avoiding some pointer dereferencing. To find out more, read the proposal on unboxed types on GitHub.

The Legacy

The Multicore OCaml project has been full of challenges, successes, and surprises. Along the way, the team has developed and grown, learning important lessons and adapted their approach to best suit the needs of all OCaml users.

Making the leap from research to product is a complex process that takes time to execute properly. In computer science, it can take decades to get right. It’s a massive achievement to get a revolutionary update like OCaml Multicore from concept to finished product in less than 8 years.

It’s also an update suitable for everyone. Users who don’t have a need for multicore features can carry on using OCaml like they always have, benefitting from other OCaml 5 features without having to change a line of code. On the other hand, the significant number of people who have long awaited the update can now benefit from having OCaml and all its strengths on multiple cores.

The story of OCaml Multicore is one of hard work and a dedication to learning. It speaks to anyone with a passion project that seems too innovative or experimental to succeed. With a strong team and a flexible, problem-solving approach, theory can quickly become reality.

Acknowledgements

A big thank you to KC Sivaramakrishnan, without whom this article would not be possible. Further thanks goes to Jerôme Vouillon and Leo White for their expertise and contributions to the ‘where do we go from here’ section of the article.

+
Contact Tarides to see how OCaml can benefit your business and/or for support while learning OCaml. Follow us on Twitter and LinkedIn to ensure you never miss a post, and join the OCaml discussion on Discuss!
+

Sources and Further Reading

+
A collection of libraries, experiments, and ideas relating to OCaml 5: https://github.com/ocaml-multicore/awesome-multicore-ocaml
+
+
A wiki for Multicore OCaml. Note that it's not currently being maintained, so whilst it has much useful information, some migh be outdated: https://github.com/ocaml-multicore/ocaml-multicore/wiki
+
+
Information on Effect Handlers: https://kcsrk.info/webman/manual/effects.html
+
+
Information on Parallelism: https://kcsrk.info/webman/manual/parallelism.html
+
+
Information on Memory Models: https://kcsrk.info/webman/manual/memorymodel.html
+
+
Academic publications pertaining to OCaml Multicore: https://github.com/ocaml-multicore/awesome-multicore-ocaml#papers
+
+
OCaml’s home on the web: https://ocaml.org
+

Lambda Retreat Report

tarides — 2023-01-12T00:00:00-00:00

Today we're taking a little pause from our OCaml 5 series to talk about a programming retreat. I spent a week in the woods with fellow programmers at the Lambda Retreat. It was a wonderful way to explore the nature of computations, abrstractions, and paradigms. Although I mostly work in OCaml, it was fun and challenging to code in Scheme, another functional programming language.

For more OCaml 5 posts, visit our Tarides blog for posts about some exciting new features and interviews with OCaml programmers, including one with me! Next week, come back to read an article about how OCaml 5 performs during the Benchmarking Game, but for now, read on for a Lambda Retreat Retrospective.

Structure and Interpretation of Computer Programs (SICP) is many programmers' favourite programming textbook. It teaches programming constructs like recursion, modularity, abstractions, etc. For a long time, it was used as the textbook for an introduction to programming course. Here's what Peter Norvig and Eli Bendersky have to say about SICP.

Having seen a lot of people highly recommend SICP, I grabbed a copy for myself a few years ago and started reading it, but, alas, I never completed the book.

In the latter part of last year, Anand decided to host a week-long retreat to gather a bunch of people and go through some interesting parts of SICP. Suffice it to say, I jumped at the opportunity to do nothing but read and write code for a week.

Getting Ready

Two weeks before the retreat, we had some warm-up sessions to get ourselves ready. During this time, we attended some remote sessions and solved a few exercises from the Chapters 1 & 2 of SICP.

Arriving

On Day 0, we all arrived at Bangalore from various parts of India, and carpooled to the Tamarind Valley Collective (TVC), located ~80km from the city. Reaching TVC turned out to be an unexpected but enjoyable 1.5km trek, since the roads to the campsite were unusable due to rain.

At the Retreat

Functional Geometry

The retreat began with Functional Geometry from the second chapter of SICP. We started with the basics, like rendering images, and slowly built the primitives needed for generating Escher's woodcut.

It was amazing to see the power of composability! We thoroughly enjoyed building Escher's woodcut from an unassuming image of a fish.

+ + + + +

+ + + + + + + + + + + + + +

Anand rewarded everyone with an Escher's woodcut T-Shirt for successfully generating the square-limit \o/

We then abstracted out the implementation details for the Functional Geometry primitives we had built. The abstraction gives us the freedom to change the implementation at a later point without affecting the higher-level details.

Mutability and State

We spent some time understanding mutations, global state, and local state in Scheme, a functional language like OCaml. This led us to building some mutable data structures, like a mutable queue and a mutable hash table in Scheme, and generalising with a dispatcher to perform operations.

Metacircular Interpreter

Another exercise before the retreat was to write a parser for Scheme in Python. At the retreat, we started with translating it to Scheme. Going further, we built a metacircular interpreter -- a Scheme interpreter written in Scheme. How cool is that?

We then learnt about lazy evaluation in Scheme and went on to make our metacircular interpreter lazy by default. Another interesting part we looked forward to was targeting WebAssembly (Wasm) from Scheme. It was surprisingly simple to go from Scheme to Wasm, targeting Wasm's Lisp-like syntax.

Beyond Tech

Living at TVC in the middle of a forest with barely any electricity or cellular network for a week was a humbling experience. Madhav, the resident manager at TVC, and his team made sure our stay was comfortable. The food, made from locally-sourced indgredients featuring local cuising was amazing!

+ + + + +

The evening walks and hikes at TVC were memorable. We managed to sight some kingfishers and owls while snacking on some freshly plucked tamarinds. We had so much fun hiking along a stream that runs in the middle of TVC and capturing wisdom about sustainable living from Madhav and Vikrant, who put it into practice by living on farms.

+ + + + + + + + + + + + + +

Our coworkers Pappu the hunting cat, her three kittens, and our boy Poco, the rugged looking sweet doggo, kept us company.

+ + + + + + + + + + + + + +

Our days of hacking were followed by board games in evenings and nights. We had so much fun and laughter riots playing games like Skull, Chameleon, and Ticket to Ride Europe. By the end of the retreat, we were surprised by how little internet and social media we had consumed that week!

+ + + + +

I'm grateful to have had the opportunity to attend the first ever Lambda Retreat and hope to carry the functional programming spirit forward. It was super nice meeting all the enthusiastic and kind people at the retreat, and I hope to see everyone again at future events.

Thanks to Anand for organising it and to everyone who attended for making it an enjoyable experience. Thanks also to Madhav and his team at TVC for ensuring our stay was comfortable.

Check out our series on Multicore OCaml, a project I've worked on for the last several years, starting with the announcement of the OCaml 5 release. If you'd like to know more about OCaml 5, you can start with KC's keynote address from the ICFP 2022 conference, OCaml tutorials, and the informative book Real World OCaml.

+
Contact Tarides to see how OCaml can benefit your business and/or for support while learning OCaml. Follow us on Twitter and LinkedIn to ensure you never miss a post, and join the OCaml discussion on Discuss!
+

Engineer Spotlight: Sudha Parimala

tarides — 2023-01-10T00:00:00-00:00

For our third and final Engineer Spotlight, we interviewed Sudha Parimala, a Tarides engineer who works primarily on the Multicore Applications team. She talks about what lead her to become an OCaml programmer and why she's excited about OCaml 5, as our blog series on Multicore OCaml continues.

Christine: Why did you decide to become an OCaml programmer rather than Python or C++?

Sudha: My programming journey started with Python in high school. At that point I didn't really know much about programming, and I picked it only as an alternative to studying biology. Python's human language-like syntax made it easy to grasp the concepts as a novice, while also getting a feel of programming constructs. The education board decided to switch to C++, and I ended up learning OOP as a result. I continued learning C, Java, and such during my undergrad.

Then I discovered Haskell and was hooked. I found it wild that I could write a 200+ lines Java program with just 10 lines in Haskell. I got an opportunity to participate in a summer school organised by ACM India on Programming Language Design. This deepened my interest in Functional Programming (FP). After graduating, I got an opportunity to join KC's Multicore OCaml team at IIT Madras. I started learning OCaml then, and there's no looking back.

C: What do you like best in OCaml?

S: I like OCaml's features combined with its practicality. When I started learning OCaml, I could relate to a lot of general FP concepts I had learnt through other languages. At the same time one can easily do imperative programming (mutations) or I/O with ease.

C: What’s the coolest thing you've made with OCaml?

S: When I was working on the Multicore OCaml compiler, I found it really cool that we could easily connect OCaml directly with C, with the type safety of OCaml. If I may add a futuristic take on this is, I'd find it really cool to do hobby projects of mine with the entire stack - from web scraping, to talking to databases, to creating web apps - in OCaml.

C: Why should engineers learn OCaml?

S: I'd recommend learning OCaml to anyone curious to learn new and succint ways of expressing programs. OCaml definitely changes the way you think about programs, and I'm sure that reflects on how you write programs, in OCaml and elsewhere.

C: What are you most excited about in OCaml 5?

S: I'm really excited to see the OCaml world transition to Multicore. It will be a challenging, yet rewarding journey. Challenging because OCaml programs for more than two decades have been designed for single core. Rewarding, thanks to blazing fast performance time and direct style concurrency. The Multicore and OCaml development teams have invested time in ensuring backwards compatibility, which will hopefully ease the process a bit.

Thank you so much, Sudha, for taking the time to answer these questions about your experience with OCaml. Also read Zach Shipko's and Jules Aguillon's interviews for their take! Thanks to their willingness to share, other developers can see why they should learn OCaml as their next language.

Feel like learning OCaml? Get started with the tutorials and the Real World OCaml book. Get a nice overview of what OCaml 5 has to offer by watching KC Sivaramamakrishnan's keynote address.

+
Contact Tarides to see how OCaml can benefit your business and/or for support while learning OCaml. Follow us on Twitter and LinkedIn to ensure you never miss a post, and join the OCaml discussion on Discuss!
+

Engineer Spotlight: Zach Shipko

tarides — 2023-01-05T00:00:00-00:00

Tarides engineer Zach Shipko answers a few questions about why he decided to learn OCaml and why he's particularly excited about the OCaml 5 release. In celebration of OCaml 5, we've interviewed several engineers about their personal experience with the language and what features they enjoy. It's a great way to get some unique insight into the language from someone who works with it on a daily basis.

Christine: Why did you decide to become an OCaml programmer rather than Python or C++?

Zach: I don't really see myself as an "OCaml programmer" because I use Python, C, Rust, Javascript, and other languages quite frequently. It's my interest in many different programming languages that led me to OCaml!

C: What do you like best in OCaml?

Z: One of my favorite things about OCaml is the amount of thought put into new language features. Because of this I think the whole community values the importance of API design and correctness.

C: What’s the coolest thing you've made with OCaml?

Z: I have been working on libirmin, which provides C bindings to the Irmin API, making it possible to use Irmin directly from C and other languages. This uses Cstubs_inverted to wrap OCaml code in C functions. I don't know how "cool" that is, but everytime it works I am pleasantly surprised.

C: Why should engineers learn OCaml?

Z: Learning a new programming language can help you see problems from a new perspective and it gives you another tool to reach for when needed. OCaml has lots of nice features (pattern matching, functors, ...) that make solving certain problems more fun.

C: What are you most excited about in OCaml 5?

Z: Other than being able to use multiple cores, I am very excited about Effects (and eventually typed effects). It is an entirely new paradigm for writing applications with a lot of research behind it. To have a usable effects system in a general-purpose language like OCaml is a huge accomplishment!

Zach emphasises the importance of expanding your horizons as a programmer, learning new languages to give you fresh perspectives and insights. Perhaps especially when learning a language like OCaml. Like the esteemed Alan Perlis said, "A language that doesn't affect the way you think about programming is not worth knowing."

Zach also studied photography and digital media in college. He took the pictures at the top of this post and said he "typically picks photos like this to have some nature to look at on programming websites."

Read Jules Aguillon's interview from 27 December 2022 to learn about his journey to OCaml. Next week, look for our final Engineer Spotlight interview with Sudha Parimala, as well as a post from her on the Benchmarking Game.

Feel like learning OCaml? Get started with the tutorials and the Real World OCaml book. Learn more about effects and other things OCaml 5 has to offer? Watch KC Sivaramamakrishnan keynote and check out the speaker deck for his talk as well.

+
Contact Tarides to see how OCaml can benefit your business and/or for support while learning OCaml. Follow us on Twitter and LinkedIn to ensure you never miss a post, and join the OCaml discussion on Discuss!
+

Engineer Spotlight: Jules Aguillon

tarides — 2022-12-29T00:00:00-00:00

In celebration of the OCaml 5 release, we decided to interview a few of our talented engineers about OCaml. While it isn't a well-known language outside of the functional programming community, we're striving to get the word out about the great benefits of OCaml and why it's worth your time to try it out, especially with the introduction of Multicore support in OCaml 5.

KC Sivaramakrishnan's inspiring keynote address is a great introduction to OCaml 5 and all it offers. Check out the speaker deck as well.

Today our engineer Jules Aguillon, who works primarily on our OCaml Platform tooling project, talks about his journey to OCaml, what he enjoys about the language, and why he thinks you should learn it! Take it away Jules!

Christine: How did you start programming?

Jules: My programming journey began with learning C in school, but I soon realised I wanted more abstraction. It felt like the C code was always talking about low-level stuff instead of what I wanted to express. In C, translations are a lot of work and get harder as the algorithm becomes more complex. Things like ASTs and polymorphic datastructures are also really hard to write in C.

Some of the ways C works are not so innocent, it supports the kinds of dangerous memory operations that famously cause 70% of all security bugs in some big corporations.

C: What did you do to get that abstraction that you were looking for?

J: I decided to learn C++, which is C but with classes (grouping code and data together), abstracted memory operations (making them safer by default), and more type checking.

But I quickly found that C++ also wasn't a good fit, mainly due to its use of boilerplate. Boilerplate refers to pieces of code that must be repeated in various places without significant change, wrapping around every concept you try to express in C++. They are used to represent several complicated concepts, and any mistake in the boilerplate could bring things like memory unsafety back. I wanted to abstract this away, too.

C: What did you do next?

J: To finally write shorter and safe code, I tried Python. It was a joy to use compared to the previous unsafe and verbose C and C++. The garbage collector solved the memory-unsafety problems, and the built-in datastructures and idioms allowed me to write many complex algorithms using only a small amount of code.

But Python has a dark side: it entirely lacks static type checking. This means that it requires considerable effort to find a type-related mistake. The only way is to run the program with different inputs and wait until it crashes. This gets really annoying as the program grows.

Furthermore, this kind of mistake happens all the time (sometimes once in every line of code!) and could be entirely solved by a type checker.

+
"For me, this is already the perfect language and it doesn't stop there!"
+

C: Is this where OCaml comes in?

J: Yes! Then I learned OCaml! It's unconditionally memory-safe, has a garbage collector, the code is concise, many kinds of abstractions are possible, and most importantly, it has well-defined and powerful type checking.

For me, this is already the perfect language and it doesn't stop there! Modules, polymorphism, and higher-order functions all add deep abstraction possibilities, and there's even a more important feature. Variant types allow types to have different shapes and write tree-looking things like ASTs (abstract syntax trees) that are impossibly hard to express in all the languages above and many others that I have tried.

Theoreticians talk about algebra of types, and this is the "plus" operation. Now that I've used it in OCaml, I could never go back to a language that doesn't have the "plus" operation!

A big thank you to Jules for taking the time to speak about his experience with OCaml. Getting a personal account of why he chose OCaml gives great insight into the strengths and features of the language from someone who uses it every day.

If you're interested in learning more about OCaml you can learn from tutorials, the Real World OCaml book, and contribute on Github. We look forward to seeing you in the community!

Love Rust? Then OCaml's New Eio Library is for You

tarides — 2022-12-27T00:00:00-00:00

We’ve come to expect a lot from the programming languages we use. We want the memory safety of Java, the performance of C/C++, and the concurrency of Go. On top of this, we need robust cybersecurity tools to protect us from the many risks and vulnerabilities in the world, all in an intuitive and easy-to-use package for programmers.

You can expect all of the above with OCaml 5. The new library Eio introduces some great new features that let the programmer write concurrent code in a way that best suits them. Eio is fast, solves the function colouring problem, and can use effect handlers to let the developer customise the scheduling algorithm rather than baking it in at runtime. If you’re a fan of how Rust delivers fast and high performing concurrent code, OCaml 5’s Eio is a close match, with some additional features.

Eio gives OCaml a new edge on speed, ease-of-use, portability, and security. It matches Rust’s reputed performance on the same points, making the two languages more comparable when it comes to concurrent programming. Let me know what you think of my comparison on Discuss or Twitter.

How Eio Makes Concurrent Code Quick and Easy

Rust is a popular programming language that solves a lot of problems for programmers. One of Rust’s strengths is the way it delivers concurrent code in a quick and safe manner.

According to a recent blog post, “Rust solves problems that C/C++ developers have been struggling with for a long time: memory errors and concurrent programming. This is seen as its main benefit.”

Eio makes writing concurrent code in OCaml much easier, resolving earlier pain points and providing significant benefits. OCaml is also a type- and memory-safe language with a low-latency and high-throughput concurrent garbage collector that doesn't get in the way of application code execution.

Below is an overview of the biggest changes Eio brings to concurrent programming in OCaml. With these improvements, OCaml provides a concurrent programming experience comparable to Rust.

Key Benefits of Eio

Performance +Eio brings big performance improvements to concurrent code in OCaml, making use cases like web servers serving requests from users a lot faster. In a speed test comparing Eio’s performance to Go’s net/http and Rust’s hyper, the results show that Eio outperforms Go and closely matches Rust. Eio can reliably serve over one million requests per second on a few cores. Being such a close match in terms of performance, OCaml is a strong contender for users looking to expand beyond using Rust.

+ + + + +

Ease-of-Use +One of Rust’s strengths is its user friendliness, meaning it’s easy to use and code in. With the OCaml 5 update, Eio makes OCaml a much easier language for writing concurrent code.

Eio offers an alternative to monadic I/O, which used to be the only way to write concurrent code in OCaml. Now, with OCaml 5, the developer experience is greatly simplified and will feel familiar to anyone who knows OCaml.

Patrick Ferris, a developer at Tarides, emphasises that with Eio:

“Normal OCaml features just work out of the box, like exception backtraces, which are crucial for writing new libraries, using tools, and debugging programs.”

Being able to use backtraces is a big improvement, as they can show the developer a form of ‘history’ of what interventions have been made to a program. Using it in Eio lets developers debug and troubleshoot more quickly. Having access to standard OCaml features also means that the more difficult parts of concurrent programming, such as cancellations, cleaning up resources, reporting errors, and testing are a lot easier to perform with Eio.

The biggest change users will notice is the resolution of the ‘code colouring problem.’ In the past, synchronous code could not exist alongside asynchronous code without breaking, requiring the developer to use a special calling convention to invoke asynchronous code. With Eio, that is no longer a problem, as both just appear as normal OCaml functions. This significantly improves developer experience and productivity and is unique to Eio. Currently, in Rust’s I/O library the ‘code colouring problem’ still exists, and developers have to spend time resolving conflicts between code types.

Portability +Both Rust and OCaml offer excellent portability, and with Eio OCaml gets several quality-of-life updates that lets developers create programs in different environments with several different features.

Operating systems have changed a lot in the last decade, benefiting from continuous development and modernisation. Thanks to its flexibility, Eio is able to take advantage of modern OS features (such as Linux’s io_uring to boost its own performance.

In turn, different backends for various platforms (such as Linux, MacOS, Windows, Mirage, etc.) can also implement the standard environment Eio expects to run programs. This adds an element of predictability to developer workflow, minimising the amount of task-switching and time spent outside of programming.

On the topic of Eio’s flexibility, Thomas Leonard, the creator and lead maintainer of Eio, highlights that:

“Eio can also run existing Lwt and Async code alongside new code, allowing existing projects to be upgraded piece by piece, keeping the tests passing throughout the migration. A couple of lines of code is all it takes to make an existing Lwt application run on Eio, and from there any new code can use Eio directly.”

Instead of having to pick one I/O tool to learn, Eio’s library can run all three in the same program which is completely new.

Security +Both Rust and OCaml offer strong safety features. According to the blog Codilime, “High performance and safety are the features that made Rust so appealing.” Well, Eio adds even more security features to OCaml’s already long list.

Eio allows developers to implement measures with great specificity, which has great significance when it comes to security. For example, Eio lets a developer program a web server to serve files only from within specified directory trees, removing the possibility that it could be tricked into serving other files. This ensures that the I/O only shares what it’s intended to do without being bypassed, which is a common security problem with web servers.

Conclusion

Many businesses and programmers love Rust, and for good reason! What OCaml 5 and Eio can offer is an alternative that matches Rust on performance and user friendliness, includes new cutting-edge features, and delivers on safety in a way that is uniquely OCaml. With its new Eio library, concurrent programming in OCaml becomes more similar to Rust, and out of the two only OCaml solves the function colouring problem. If you’re looking to complement your use of Rust with a robust functional programming language – without sacrificing performance – OCaml 5 is the language for you.

Contact us and learn more about how OCaml can transform your business. You can also find us on GitHub, the OCaml Discuss forum, and Twitter

Acknowledgements

With thanks to Thomas Leonard, creator and lead maintainer of Eio, and Patrick Ferris, a developer at Tarides, for their expertise and input that made this article possible.

Sources

+
Codilime blog
+
+
Serokell blog
+

OCaml 5 Multicore Testing Tools

tarides — 2022-12-22T00:00:00-00:00

The new version of OCaml 5 is here! It brings the ability to program multicore applications and to maximise our usage of all the CPU cores without a global lock getting in the way of performance. What's most exciting to me though is that we have a whole new way of writing... bugs!

And with so much potential for mistakes comes a new era of testing tools to help us write correct applications:

Memory Model

The first of those is the memory model of OCaml 5. If you already know what those two words mean, please skip this part because I won't pretend that I do. (I'm still convinced that it's just some fancy legalese terms to confuse people.) But it may actually matter when you realise that you've been living in a fantasy your whole CPU life:

left := 42 ;
+right := 0 ;

When I read these two lines, I have years of beliefs telling me that the reference left will be updated before the right one is. But modern compilers and hardware conspire to break any sanity that may exist in my brain. You see, the order of operations doesn't actually matter if you can't see that the CPU is doing things in another order. It may just happen that the compiler or CPU will choose to do those two operations in reverse if they think it would be more convenient. And without this, our software would be so much slower that "instructions are executed in order" is an essential lie. (Well, is it even a lie if it makes no difference?)

To catch a liar, you need a second observer to correlate their claims. This is exactly what the other CPU cores will do. The bad news is that they are not actively looking for bad behavior from their colleagues, but they will end up reading values that aren't quite written in the order expected. This will wreak havoc into your invariants and trigger very, very weird bugs.

I'm not kidding when I say "very, very weird." Below is a real example of an out-of-order read/write that happened on my computer. This was a very simple program, with only two references, left and right, that got updated by two different domains (shown as two branches here):

                          !left    = 42
+                          !right   = 0
+                                 |
+              .------------------------------------.
+              |                                    |
+              |                               left  := 1
+              |                               right := 2
+         right := 3                                |
+        !left        = 42                    !right       = 3
+                      ^^^^
+                      how?

I tried to align the sequence of operations according to the observed memory values, but no ordering actually made sense. We can't have both !left = 42 and !right = 3 in the end.

Here's another attempt to align the instructions in a coherent way:

                          !left    = 42
+                          !right   = 0
+                                 |
+              .------------------------------------.
+              |                                    |
+         right := 3                                |
+        !left        = 42                          |
+                                              left  := 1
+                                              right := 2
+                                             !right       = 3
+                                                          ^^^^
+                                                          how?

It already requires some time to unpack this short example, but imagine how bad it would get to debug such a thing in production!

I want to stress that this specific execution wasn't the result of a compiler optimisation that we could have discovered by reading the assembly code. The program was running just fine over many iterations before being disturbed by a sudden hardware optimisation. The probability of observing this behavior from your CPU is very low---not low enough that you can ignore it, but you won't be able to reproduce this exact bug in any reasonable time. (But we'll see how to catch our CPU cores red-handed in the following sections!)

But ok, wait---come again. How is any of this nonsense a good memory model? For starters, the values you can read "out-of-order" are still real values that have been assigned to the references, not imaginary ones. Yes, it could be even weirder, but you don't want to know. It's all fun and games with integers, but this property really matters for pointers (where following the wrong one would lead you down a segmentation fault). You need to be wary of this in other languages, but not in OCaml. Memory safety is preserved. It's not an instant Game Over to do an accidental out-of-order read.

Secondly, when reading and writing to shared memory, you should use the new Atomic module to ensure the proper memory ordering of operations. This will introduce the required memory barriers to bring back sanity---at a small performance cost---so it's opt-in and only required for shared memory! (Note: you can also use a Mutex lock to protect your read/write into shared memory.)

In technical words, OCaml 5 programs enjoy the "Sequential Consistency for Data Race Freedom (DRF-SC)" property. If your program has no data races, then you can reason about your code under sequential consistency where the operations from different threads are interleaved with each other, but the instructions don't seem to be executed out of order.

By using Atomic, we are back in the wonderful land where operations happen in the expected order! The memory model becomes a tool for your brain, enabling it to reason about your algorithms. This one is so intuitive that I can once again pretend that it doesn't exist (without getting hit by an unexpected bug later.)

ThreadSanitizer

Alright, so how do we check that our programs aren't susceptible to an "out-of-order" bug caused by a missing Atomic or Mutex? ThreadSanitizer was created by Google as a lightweight instrumentation to discover these runtime data races.

To enable it on your OCaml program, you’ll need a special compiler that adds the required instrumentation to your software. Don't worry, it’s super easy to setup thanks to opam switches!

Install and usage informations on the ocaml-tsan repository

As a running example to demonstrate the usefulness of each tool, let's look at different implementations of simple banking accounts, where users can transfer money to each other if they have enough money in their account:

module Bank = struct
+  type t = int array
+
+  let transfer t from_account to_account money =
+    if money > 0                     (* no negative transfer! *)
+    && from_account <> to_account    (* or transfer to self! *)
+    && t.(from_account) >= money     (* and you must have enough money! *)
+    then begin
+      t.(from_account) <- t.(from_account) - money ;
+      t.(to_account)   <- t.(to_account)   + money ;
+    end
+end

This module could be part of a much larger program that receives transaction requests from the network and handles them. For simplicity here, we'll only be running a small simulation, but ThreadSanitizer is intended to be used on large real programs with messy I/O and side effects, not just broken toys and unit tests.

let t : Bank.t = Array.make 8 100  (* 8 accounts with $100 each *)
+
+let money_shuffle () = (* simulate an economy *)
+  for _ = 0 to 10 do
+    Unix.sleepf 0.1 ; (* wait for a network request *)
+    Bank.transfer t (Random.int 8) (Random.int 8) 1 ; (* transfer $1 *)
+  done
+
+let account_balances () = (* inspect the bank accounts *)
+  for _ = 0 to 10 do
+    Array.iter (Format.printf "%i ") t ; Format.printf "@." ;
+    Unix.sleepf 0.1 ;
+  done
+
+let () = (* run the simulation and the debug view in parallel *)
+  [| Domain.spawn money_shuffle ; Domain.spawn account_balances |] 
+  |> Array.iter Domain.join

It should be pretty clear that our code is not thread-safe and that transferring money while printing the account balances is asking for trouble! Running it with ThreadSanitizer enabled will print warnings into the terminal as soon as a potential data-race is observed (and it's even better in real life as the output is colorful!):

WARNING: ThreadSanitizer: data race (pid=1178477)
+  Write of size 8 at 0x7fc4936fd6b0 by thread T4 (mutexes: write M87):
+    #0 camlDune__exe__V0.transfer_317 <null> (v0.exe+0x6ae1a)
+    #1 camlDune__exe__V0.money_shuffle_325 <null> (v0.exe+0x6af8d)
+    .. ...
+
+  Previous read of size 8 at 0x7fc4936fd6b0 by thread T1 (mutexes: write M83):
+    #0 camlStdlib__Array.iter_329 <null> (v0.exe+0x9c675)
+    #1 camlDune__exe__V0.account_balances_563 <null> (v0.exe+0x6b054)
+    .. ...

The issue is reported very clearly thanks to the two conflicting stacktraces. There's a read/write data-race happening between the money_shuffle execution and the account_balances one, which could result in unreasonable memory reordering artifacts. In fact, it would be even worse if we were to transfer money from multiple domains in parallel (which we'll attempt to do in the next section as an interesting way of speeding up our bank transactions with Multicore).

It looks like we can fix the read/write data-race by adding a Mutex lock around the transfer function write operations:

let lock = Mutex.create ()
+
+let transfer t from_account to_account money =
+  Mutex.lock lock ;
+  (* ... same as before ... *)
+  Mutex.unlock lock

But ThreadSanitizer is not easily fooled and will still complain loudly. We also need to use the same Mutex to protect the array reads in the account_balances function, as it would otherwise be perfectly valid to optimise away the shared memory reads into oblivion:

let account_balances_optimized () = (* faster... but wrong-er! *)
+  let str = String.concat " " @@ Array.of_list @@ Array.map string_of_int t in
+  for _ = 0 to 10 do
+    Format.printf "%s @." str ;
+    Unix.sleepf 0.1 ;
+  done

The data races reported by ThreadSanitizer are not only the ones where an absurd “out-of-order” happened, but preemptively, all those that could potentially trigger such a problem. If you are porting an existing multi-threaded application to OCaml 5, this compiler variant should probably be your default debug build.

Note that the ThreadSanitizer instrumentation does add a performance cost and doesn't increase memory safety by itself. Run it for a bit, track down your shared memory misuses, and add the required Atomic and Mutex operations.

Multicore Tests: `Lin` (and `STM`)

How do we unit test our Multicore libraries? It's business as usual, and the standard Alcotest will do well, for example. But there are some new properties that we should look for when writing and using libraries in a multicore setting. Let's revisit the bank accounts implementation by using Atomic operations this time rather than a Mutex:

module Bank = struct
+  type t = int Atomic.t array
+
+  let get t client = Atomic.get t.(client)
+
+  let transfer t from_account to_account money =
+    if money > 0
+    && from_account <> to_account
+    && get t from_account >= money
+    then begin
+      (* [fetch_and_add x v] is an atomic operation that does [x := !x + v] *)
+      let _ : int = Atomic.fetch_and_add t.(from_account) (- money) in
+      let _ : int = Atomic.fetch_and_add t.(to_account) money in
+      ()
+    end
+end

See how careful I was to use Atomic to read and write from shared memory? (I even used a fancy fetch_and_add!) Therefore, it must be correct if used by different domains, right? While this program doesn't have a data race, the definition of "correctness" is more subtle in a multicore setting.

It's easier to explain if I show you the problem. To test this interface with the Lin library, we only need to describe how to initialise a new bank and the API signature of available functions:

open Lin
+
+module Bank_test = struct
+  type t = Bank.t
+
+  let init () = (* 8 accounts with $100 each *)
+    Array.init 8 (fun _ -> Atomic.make 100)
+  let cleanup _ = ()
+
+  let account = int_bound 7 (* array index between 0..7 *)
+  let api = [
+    val_ "get" Bank.get (t @-> account @-> returning int) ;
+    val_ "transfer" Bank.transfer
+      (t @-> account @-> account @-> nat_small @-> returning unit) ;
+  ]
+end
+
+module Run = Lin_domain.Make (Bank_test)
+
+let () =
+  QCheck_base_runner.run_tests_main [Run.lin_test ~count:1000 ~name:"Bank"]

That's all! A small DSL to define the types of our functions and we are done. Run this test to admire the beautiful ASCII art... craAaAash:

  Results incompatible with sequential execution
+
+                                   |
+                                   |
+                .------------------------------------.
+                |                                    |
+     transfer t 4 0 8  : ()               transfer t 4 0 93  : ()
+          get t 4  : -1

Lin found a bug! The account number 4 is trying to simultaneously transfer $8 and $93 to account number 0. It then naturally ends up with a negative $1 on its account, which is obviously bad for a banking system... but we never told Lin that this was illegal, so why is it complaining?

In the tradition of QuickCheck, Lin not only generates random arguments to test our API, but it also creates full programs to execute on two domains. It then runs these generated programs and checks if the intermediate results are "sequentially consistent," the property of a well-behaved API where we can always explain its multicore behavior as a linear execution of the calls on a single core.

Without this "sequential consistency" property, the internals of our functions leak when multiple cores interleave their execution. In the example above, it wouldn't be possible to reach a negative $1 account on a single core, so Lin reports that sequential consistency is broken. It doesn't know that negative accounts are illegal, but it knows that this state is unreachable without multicore shenanigans.

+ + + + +

Even though this is not a data race, a user of our library would have an equally hard time understanding the outcomes of our functions, when they depend so much on their accidental interleaving. "It sometimes doesn't work" is not a bug report I wish to see!

An intuitive way of thinking about "sequential consistency" is that our functions should behave as if they were a single atomic operation: Either we see none of their side effects, or we see all of them. It shouldn't be possible to see an in between, as this would result in a non-sequentialisable execution.

Once again, the easiest solution here is to use a Mutex to lock all the accounts during a transfer and when reading an account balance. Run the test suite again with Lin, and yep, we are safe. The operations are now sequentially consistent! (But we don't need the Atomics anymore with the Mutex.)

+ + + + +

I really like how low effort / high reward Lin is. In just a few lines of declarative code, we can check that our code is correct when running on multiple cores. It's very extensive in its testing, which is just what we need when bugs are this hard to reproduce. The Multicore testing suite also provides a state-machine interface STM, which allows you to specify more properties that your system should respect (not only sequential consistency, but custom business logic!)

More examples on the multicoretests repository

Fun fact: The earlier "out-of-order" memory read/write on mutable references was also generated by Lin. While this tool is not specialised like ThreadSanitizer for discovering data-races, it can still trigger and identify the hardware memory reordering since they produce outcome that can't be explained on a single core. Here's the complete test if you want to see your computer memory ~~misbehaving~~ optimising:

open Lin
+
+module Int_array = struct
+  type t = int array
+  let init () = [| 0 ; 0 |]
+  let cleanup _ = ()
+  let index = int_bound 1
+  let api = [
+    val_ "get" Array.get (t @-> index @-> returning int) ;
+    val_ "set" Array.set (t @-> index @-> int @-> returning unit) ;
+  ]
+end
+
+module Run = Lin_domain.Make (Int_array)
+let () = QCheck_base_runner.run_tests_main [Run.lin_test ~count:10_000 ~name:"Array"]

Dscheck

While adding a Mutex restores the sequential consistency of the transfer function, it's unsatisfying to slow down all transactions with a global lock. Most transfers are going to happen on different accounts, so could we be more precise in our safety measures? This is far from easy with locks, if we don't want to end up bankrupt with hungry philosophers!

One alternative solution is called lock-free programming, and as the name implies, it gets rid of all the locks---but at the cost of more complex algorithms. By using only Atomic operations, there are ways of encoding our transfer operation without blocking the other cores (such that they don't get stuck waiting on the unrelated threads to finish their transaction).

Lock-free algorithms have a bad reputation of being crazy hard to implement correctly. It's very easy to convince yourself that you found the right solution, only to discover that your algorithm only works when the OS scheduler is on your side (which it generally is...until it isn't). This is another type of hard-to-reproduce bug. We can't coerce the OS scheduler to be evil when testing our software.

Our last testing tool is the library dscheck. It provides a way to exhaustively test all the possible schedulings of a Multicore execution in order to discover the worst-case scenario that would lead to a crash. It does so by simulating parallelism on a single core using concurrency, thanks to algebraic effects and a custom scheduler. Dscheck is very fast because it doesn't test all possible interleaving but only the ones that matters.

In order to use it, you only need to replace the Atomic module by a custom one, and then write your unit test. Here I simply copy-pasted the bug generated by Lin earlier:

module Atomic = Dscheck.TracedAtomic
+module Test   = Dscheck.TracedAtomic
+
+module Bank = struct (* same as before, but now using the traced Atomic *) end
+
+let test () =
+  let t = Array.init 2 (fun _ -> Atomic.make 100) in
+  Test.spawn (fun () -> Bank.transfer t 0 1 8) ; (* fake a Domain.spawn *)
+  Bank.transfer t 0 1 93 ;
+  assert (Bank.get t 0 >= 0)
+
+let () = Test.trace test (* exhaustively test all interleaving *)

Dscheck will then run our test function multiple times, discovering all the interesting paths that the scheduler could lead us down, and finally outputs a visualisation describing the worst-case scheduling that lead to crashes:

+ + + + +

This is a low-level view of the bug. By inspecting the sequence of Atomic operations along the bad paths, we can discover the origin of the problem: the code is not careful when removing money from an account.

Perhaps this would work better:

let rec transfer t from_account to_account money =
+  let money_from = get t from_account in
+  if money > 0
+  && from_account <> to_account
+  && money_from >= money
+  then begin
+    if Atomic.compare_and_set t.(from_account) money_from (money_from - money)
+    then let _ : int = Atomic.fetch_and_add t.(to_account) money in ()
+    else transfer t from_account to_account money (* retry *)
+  end

And yes, Dscheck now happily validates all possible interleaving of this unit test!

More examples on the dscheck repository

So, does it mean our banking system works now? Nope! Lin reports new counter examples that break sequential consistency. I told you that lock-free was hard! Still, we can keep iterating, and we will eventually get it right because our tools remove the doubt and the impossibility of reproducibility that would otherwise make the task insurmountable. It's like having tiny assistants to double-check our assumptions. I love it. I've never been so excited to test my software!

                           get t 3  : 100
+                           get t 4  : 100
+                                   |
+                .------------------------------------.
+                |                                    |
+                |                             transfer t 4 3 10  : ()
+           get t 4  : 90
+           get t 3  : 100
+                     ^^^^^
+                how? account 4 sent the money, but account 3 didn't receive it!

Would you have caught this bug? Can you fix it? ;)

This was a tiny example, and it already brought some surprises. The multicore testing Lin, STM, and dscheck have been applied to real datastructures with great success. In fact, I wouldn't trust lock-free algorithms that were not validated by them.

Conclusion

It's 2022 and OCaml is finally Multicore. Even if this article is only scratching the surface of a specific itch, I hope it has convinced you that the Multicore metamorphose wasn't only about lifting a global lock somewhere in the runtime. A lot of care and attention also went into creating a great environment to tackle really hard problems. Here we've only looked at:

The memory model to be able to reason about our programs
ThreadSanitizer to detect dangerous use of shared memory
Lin and STM to discover logical bugs in a multicore setting
Dscheck to validate unit tests of lock-free algorithms by exhaustively checking all possible interleavings of their Atomic operations

There's still a lot more to discover in the latest release of OCaml. In the meantime, not only can we do Multicore, we can do it with confidence that our code works!

Advanced Merlin Features: Destruct and Construct

tarides — 2022-12-21T00:00:00-00:00

Merlin is one of the most important tools for OCaml users, but a lot of its +advanced feature often remain unknown. For OCaml newcomers who might not know, Merlin is the server software that provides intelligence to code editors when working on OCaml documents. It allows one to easily navigate the code, get meaningful information (like type information), and perform code generation and refactoring tasks. Merlin installation and usage is documented on its official webpage.

Merlin is distributed with both an Emacs and a Vim plugin. It can also be used in Vscode via the OCaml LSP Server and the corresponding plugin.

In this post, we will focus on two complementary features of Merlin: the venerable destruct and the younger construct. Both of these leverage OCaml's precise type information to destruct or create expressions.

Destruct

Destruct (sometimes called case-analysis) uses the type of an identifier to +perform multiple tasks related to pattern-matching. It can be called with the +following key bindings:

Emacs: C-d or M-x merlin-destruct
Vim: :MerlinDestruct
VSCode: Alt-d or 💡 Destruct

Destruct's behavior changes slightly depending on the context around the cursor. We are going to describe how it behaves in the next three sections.

Automatic Case Analysis

The primary use case for Destruct is to generate a pattern-matching for a +given value. Let's consider the following snippet:

let f (x : int option) = x

Calling destruct on the right-most occurrence of x will automatically generate the following pattern-matching with the two constructors of x's' option type:

let f (x : int option) = match x with
+  | None -> _
+  | Some _ -> _

What happened is that Merlin looked at the type of x and generated a complete pattern-matching by enumerating its constructors.

Notice that Merlin used underscores on the right-handsides of the matching. We call these underscores typed holes. These holes are rejected by the compiler, but Merlin will provide type information for them. These holes should not be confused with the wildcard pattern appearing on the left handside Some _.

After calling destruct, the cursor should have jumped to the first hole. In +Emacs (resp. Vim), you can navigate between holes by using the commands M-x merlin-next-hole (resp. :MerlinNextHole) and M-x merlin-previous-hole (resp. :MerlinPreviousHole). In VSCode, you can use Alt-y to jump to the next typed hole.

Complete a Matching

Merlin can also add missing branches to an incomplete matching. Given +the following snippet:

let f (x : int option) = match x with
+  | None -> _

Calling destruct with the cursor on None will make the pattern-matching +exhaustive:

let f (x : int option) = match x with
+  | None -> _
+  | Some _ -> _

Refine the Cases

Finally, Merlin can be used to make a pattern-matching more precise when called on a wildcard pattern _. Given the following snippet:

let f (x : int option opton) = match x with
+  | None -> _
+  | Some _ -> _

Calling destruct with the cursor on the _ pattern in Some _ will refine the matching:

let f (x : int option option) = match x with
+  | None -> _
+  | Some (None) | Some (Some _) -> _

Note that Destruct also works with other types, like records. Let's consider the following snippet:

type t = { a : string option }
+let f (x : t) = x

Calling destruct on the last occurrence of x will yield:

let f (x : t) = match x with 
+  | { a } -> _

And we can refine it by calling destruct again on a, etc.

let f (x : t) = match x with 
+  | { a = None } | { a = Some _ } -> _

That wraps our presentation for destruct. Generating and completing pattern- +matching cases can be very useful when working with large sum types !

Construct

Construct can be considered as the dual of Destruct, as they work +complementarily. When called over a typed-hole _, Construct will suggest +values that can fill that hole. It can be called with the following key +bindings:

Emacs: M-x merlin-construct
Vim: :MerlinConstruct
VSCode: Alt-c of 💡 Construct an expression (the cursor must be right after the _)

For example, given the following snippet:

let x : int option = _

Calling construct with the cursor on the _ typed hole will suggest the following constructions:

Some _
+None

Choosing the first one will replace the hole and place the cursor on the next hole:

let x : int option = (Some _)

Calling construct again will suggest 0 and result in:

let x : int option = (Some 0)

In the future, Construct might also suggest fitting values from the local +environment instead of solely rely on a type's constructors.

Destruct and Construct

As stated previously calls to destruct and construct can be used in +collaboration. For example, after calling destruct on x in the following code snippet:

type t = { a : unit; b : string option }
+let f (x : int option) : t option = x

x is replaced by a matching on x with the cursor on the first hole:

let f (x : int option) : t option = 
+    match w with
+    | None -> _
+    | Some _ -> _

One can immediately call construct and choose a construction for the first branch:

let f (x : int option) : t option = 
+    match w with
+    | None -> None
+    | Some _ -> _

And again for the second branch:

let f (x : int option) : t option = 
+    match w with
+    | None -> None
+    | Some _ -> Some _

Finally, like Destruct, Construct also works with records and most OCaml types:

Some _ → Some { a = _; b = _ } → Some { a = (); b = None }

Conclusion

When put to good use, these complementary features can remove some of the burden of working with big variant types. We encourage you to try them and see if they help your everyday workflow! If you encounter any issues or have ideas for improvement, please communicate them to us via the issue tracker.

How Nomadic Labs Used Multicore Processing to Create a Faster Blockchain

tarides — 2022-12-20T00:00:00-00:00

The technology that makes blockchain possible is complex, cutting-edge, and fascinating. Balancing efficiency and security on a knife's edge, it finds perfect harmony between high transaction speeds and safe, predictable results. Blockchain technology is constantly evolving, pushing the boundaries of what’s possible, and driving innovation and research. Discover how Tarides helped Nomadic Labs use OCaml 5 to boost blockchain performance with rollup technology.

In order to remain competitive with other major players in the blockchain market (including Bitcoin and Ethereum), the open-source blockchain Tezos invests in cutting-edge research and engineering. Another way Tezos has chosen to differentiate itself in a crowded market is by focusing on sustainability and scalable efficiency. Tezos is proof-of-stake (rather than proof-of-work) and is therefore more energy efficient than other blockchain technologies, with the annual energy consumption estimated at 0.001TWh, or “17 global citizens.” Furthermore, the Cambridge Centre for Carbon Credits (4C) is using the Tezos blockchain to create a trusted decentralised marketplace for carbon credits. The sheer amount of data needed to support verifiable carbon credits requires planetary-scale computations, coupled with the digital permanence needed to track projects over decades and even generations.

A growing number of users and companies are bringing their projects to the Tezos blockchain. To respond to increasing demand, the network is preparing for more activity and high-throughput applications. To that end, Nomadic Labs is working on implementing leading-edge rollup technology for the Tezos blockchain. Rollups are a way of increasing throughput and blockchain speeds without compromising on decentralisation, latency, stability, security, or its resistance to censorship.

To achieve their goal of offering a major scaling solution for Tezos, Nomadic Labs called in Tarides. By leveraging the multicore capabilities of OCaml 5, the team was able to achieve significant performance boosts, making it a viable solution for increasing both blockchain speeds and transactions per second (TPS).

Marco Stronati co-leads the cryptography team at Nomadic Labs with cryptographer Marc Beunardeau. Looking back on the decision to use an early version of OCaml 5, Marco said, “We knew OCaml 5 was not ready yet, but that backward compatibility was an explicit goal of the project, so we thought we would put that to the test.”

Who Are Nomadic Labs?

Nomadic Labs is one of the largest research and development centres within the open-source Tezos ecosystem. They work on the Tezos core technologies that run its distributive network as one of the largest research and development centres within the Tezos ecosystem.

Nomadic Labs handles software releases and amendments to the Tezos blockchain, focusing on the innovation, development, and implementation of new features. They help companies and institutions use Tezos to their advantage. At its core, the company values dependability, balancing cutting-edge innovation with reliable and consistent results.

What Is a Rollup and How Does it Boost the Speed of the Blockchain?

Rollups settle transactions outside the network and then post data back into it. This is what’s called a Layer 2 solution, which avoids the main chain. By handling the process externally, they reduce the strain on the network. Rollups help blockchains like Tezos keep transaction speeds and throughput high without compromising the integrity of the blockchain. It is one of the many different scalability solutions a blockchain may employ to offer top-level performance. There are two main types of rollups: optimistic and zero-knowledge (zk).

Optimistic rollups assume that the transaction data they’re processing is correct, and any fraud or other problem with the transaction is handled separately. Zk-rollups on the other hand use zero-knowledge proofs to validate a transaction. Zk-rollups provide better security and confidentiality than optimistic rollups, but they come with their own sets of limitations.

Pushing the Limits of Zero-Knowledge Proving Systems

The challenge Nomadic Labs faced was related to their proving system. In general, proving systems have to be very asymmetrical, with a prover doing most of the work off-chain and a lean verifier working on-chain. Whilst these rollup systems are great for blockchains, because of how quickly they can verify something, the proving stage is very CPU- and memory-intensive. This prevents the scalability of Epoxy, their zk-rollup solution, since its throughput and latency are directly limited by the speed of the prover. +Most of the innovation in the field of zero-knowledge proving systems today is driven by lowering the complexity of the provers, thanks to novel cryptography but also to highly optimised implementations.

OCaml 5 Multicore Saves the Day

The team’s solution to the prover’s inefficiencies was to use the aPlonK proving system to power Epoxy. It enables the efficient aggregation of multiple proofs that can be executed in parallel. This is where OCaml 5 comes in! With its Multicore capabilities and strong safety features, it is the perfect candidate for speeding up the proving process.

With OCaml 5, the team could parallelise the proving process by utilising multiple cores on one machine. According to Marco, “OCaml 5 drastically improved performance with minimal effort.” Increased performance of the zk-rollup translates to high TPS and throughput for customers using digital currencies. Striving to be the fastest blockchain means constantly looking at opportunities to improve performance, from changing algorithms to workflows.

Furthermore, speaking as a seasoned developer, Marco emphasises the importance of OCaml 5 being easy to install and set up.

“The most important thing was not having to revolutionise what I do. People don’t want to waste a week on upgrading, and this was a seamless experience.”

Installing OCaml 5 proved to be easy for the team, and in the matter of a few hours, they were running Multicore on their machines. For people who don’t need Multicore, the upgrade is completely backwards compatible, and their sequential code will still work normally. It’s due to the precise balance OCaml 5 strikes between backwards compatibility and cutting-edge upgrades that Marco thinks there is literally “No reason not to upgrade.”

When using OCaml 5 before its official launch, the team faced some small compatibility issues they needed help with. They checked the state of compatibility on the helpful health check website, where more and more packages were ‘going green’ daily. Even before they could file a bug report, they would find that their problem had been resolved. Since set up was so smooth, the team needed very little help. Still, Marco commented that: “The moment we had a problem, we would get help immediately.”

What Does the Future Hold?

The team is excited to continue using OCaml 5 in the future, as soon as Tezos begins using it in production after the full release. At the moment, the team has to parallelise using several machines to speed up the prover’s performance. With OCaml 5, they will be able to exploit multiple cores on several computers at the same time.

“We can easily double the speed of what we’re doing with OCaml 5.”

Conclusion

After successful experimentation with a prerelease of OCaml 5, the team at Nomadic Labs discovered that it gives their zk-rollup a significant performance boost. For the Tezos blockchain, this boost can result in higher TPS and throughput for customers who use their digital currency. Combined with Tezos’s other benefits, such as its energy efficiency (thanks to its proof-of-stake consensus mechanism), OCaml 5 definitively gives it a leg up with fast zk-rollups.

The technologies that make blockchains a reality are undeniably a driving force behind great innovation. Zk-rollups are just one example of technologies that aim to make complex processes like verifiers and provers lightning fast. Performance is vital in almost every field, and the use cases for this type of technology are endless.

Tarides offers its extensive expertise in OCaml to help businesses achieve their targets. Find out more about how OCaml 5 can help you transform your business!

Sources

Kathmandu Blog on Nomadic Labs +Next Generation Rollups +Ethereum Rollups +Smart Rollups

OCaml 5 With Multicore Support Is Here!

tarides — 2022-12-19T00:00:00-00:00

It's here! It's finally here! On Friday, 16 December 2022, the OCaml community announced the official release of Multicore OCaml! From the beginning, Tarides has been deeply involved in OCaml's evolution, so we're very proud to present OCaml 5!

Our work with the myriad of academics, industrial developers, and the entire OCaml community has been both inspiring and fulfilling. We look forward to continuing our collaboration for future iterations of OCaml. Watch KC's keynote to get a visual overview of all OCaml 5 has to offer!

About OCaml 5

OCaml is a pragmatic functional programming language. Its strength lies in the capacity to balance security, performance, and reliability. OCaml is used in both industry and academia to address problems in which a single mistake could be catastrophic, such as in finance and sensor analytics for sustainable agriculture. It's also used by millions of developers daily with Docker for Desktop.

OCaml 5 brings the long-awaited runtime support for shared memory parallelism and effect handlers. It is a major change (including the full rewrite of a new, concurrent garbage collector), but we worked hard to ensure there would be no breakage for existing OCaml users. This release combines the security and safety of OCaml with new features that bring huge performance benefits and an improved methodology for writing concurrent code.

OCaml 5 supports both the the x86-64 and ARM64 architectures, so Linux, the BSDs, macOS, and Mingw-w64 on Windows are all supported. Over the next year, the OCaml community and Tarides will restore support for most previously-supported architectures that fall outside of this range, but this doesn't mean you can't use OCaml now! OCaml 5 seeks to be completely backwards-compatible, and programs written for any version of OCaml 4 will continue to work in OCaml 5.

Multicore

With technological advances and the explosive growth of machines with more and more available cores, it's necessary for programming languages to support multicore technology. Until this new release, OCaml was single-threaded, meaning it could only utilise one core to run code. With OCaml 5, programs can now exploit multiple cores and execute processes in parallel, providing users with enhanced performance and efficiency.

Performance is key with OCaml 5 to ensure your programs run smoother, faster, and more efficient, a significant achievement in turning cutting-edge science into real-life applications and industrial-strength tools.

Multicore OCaml has been in the making for 8 years and required a full rewrite of it runtime environment, so you can believe that the OCaml community is absolutely thrilled to see this come to fruition from their years of hard work. Multicore support ensures beginners can achieve the same productivity as OCaml experts!

If you come across any unexpected behaviours that aren't covered by the few exceptions listed on the Discuss post, please report them on the OCaml issue tracker.

Eio & Concurrency

Concurrency in OCaml 5 is supported through the use of effect handlers, a new feature that enables the development of concurrent applications in a seamless fashion. The developer experience is improved, now that programmers can simply write concurrent code in the same style as non-concurrent code.

Eio, our experimental, high-performant I/O library, is an excellent example of use for effect handlers. Since we've already demonstrated that we can reach millions of requests per second while keeping simple, direct-style code, OCaml 5 is meant to be on par with Rust (and outperforms Go) for I/O heavy workloads.

+ + + + +

With Eio, asynchronous and synchronous code can be composed together naturally, solving the function colouring problem that affects many languages, including programs written in earlier versions of OCaml. This makes OCaml easier to use, even if you're new to the language.

Runtime Events

Another huge improvement with OCaml 5 is Runtime Events, a library included with OCaml 5's runtime, which efficiently processes performance data from the garbage collector (GC) and runtime. Runtime Events provides continuous monitoring of OCaml applications. Multicore programs are notoriously hard to debug and profile, so we ensured that OCaml 5 has a built-in mechanism to build great, relevant tooling. The basis of this is Runtime Events.

Read more about Runtime Events in the OCaml Manual, and watch the OCaml Workshop for a visual introduction to Runtime Events. Also check out these WIP tools: olly and meio.

Open Source and Open Arms

OCaml 5 is freely available to everyone worldwide. For installation instructions, compiler configurations, and a detailed list of changes and new features, please visit this OCaml Discuss post. While you're there, join in the conversation! We welcome you to the OCaml community with open arms! There has never been a better time to learn and use OCaml. Give it a try, and please report any problems to the OCaml issue tracker.

Read more about the journey to Multicore, Runtime Events, Eio, and more over the next six weeks in our OCaml 5 blog series. It will include several articles highlighting OCaml 5's new features, interviews with OCaml engineers, and reasons why OCaml should be the next language you learn. Follow us on Twitter and LinkedIn so you don't miss a thing!

This OCaml 5 release is the best holiday gift for developers, both experienced and those new to programming. Enjoy!

+
Tarides is always happy to discuss commercial opportunities around OCaml. Especially with the OCaml 5 release, there are many areas where we can help industrial users to adopt OCaml 5 more quickly, including training, support, custom developments, etc. Please contact us if you are interested in discussing your needs.
+

Hillingar: MirageOS Unikernels on NixOS

tarides — 2022-12-14T00:00:00-00:00

+ +

+ + + + +

NixOS allows reproducible deployments of systems by managing configuration declaratively. +MirageOS is a unikernel creation framework that creates targeted operating systems for high-level applications that can run on a hypervisor. +By building MirageOS unikernels with Nix, we can enable reproducible builds of these unikernels and enable easy deployment on NixOS systems.

Introduction

The Domain Name System (DNS) is a critical component of the modern Internet, allowing domain names to be mapped to IP addresses, mailservers, and more. +This allows users to access services independent of their location in the Internet using human-readable names. +We can host a DNS server ourselves to have authoritative control over our domain, protect the privacy of those using our server, increase reliability by not relying on a third party DNS provider, and allow greater customisation of the records served. +However, it can be quite challenging to deploy one's own server reliably and reproducibly. +The Nix deployment system aims to address this. +With a NixOS machine, deploying a DNS server is as simple as:

{
+  services.bind = {
+    enable = true;
+    zones."freumh.org" = {
+      master = true;
+      file = "freumh.org.zone";
+    };
+  };
+}

Which we can then query with:

$ dig freumh.org @ns1.freumh.org +short
+135.181.100.27

To enable the user to query our domain without specifying the nameserver, we have to create a glue record with our registrar pointing ns1.freumh.org to the IP address of our DNS-hosting machine.

You might notice this configuration is running the venerable bind¹, which is written in C. +As an alternative, using functional, high-level, type-safe programming languages to create network applications can greatly benefit safety and usability whilst maintaining performant execution. +One such language is OCaml.

MirageOS² is a deployment method for these OCaml programs. +Instead of running them as a traditional Unix process, we instead create a specialised 'unikernel' operating system to run the application. +They offer reduced image sizes through dead code elimination, as well as improved security and efficiency.

However, to deploy a Mirage unikernel with NixOS, one must use the imperative deployment methodologies native to the OCaml ecosystem, thus eliminating the benefit of reproducible systems that Nix offers. +This blog post will explore how we enabled reproducible deployments of Mirage unikernels by building them with Nix.

MirageOS

+ +

MirageOS is a library operating system that allows users to create unikernels, which are specialised operating systems that include both low-level operating system code and high-level application code in a single kernel and a single address space. +It was the first such 'unikernel creation framework', but comes from a long lineage of OS research, such as the exokernel library OS architecture. +Embedding application code in the kernel allows for dead-code elimination, removing OS interfaces that are unused, which reduces the unikernel's attack surface and offers improved efficiency.

+ +

Contrasting software layers in existing VM appliances vs. unikernel's standalone kernel compilation approach [3]

Mirage unikernels are written OCaml⁴. +OCaml is more practical for systems programming than other functional programming languages, such as Haskell. +It supports falling back on impure imperative code or mutable variables when warranted.

Nix

+ +

Nix snowflake⁵.

Nix is a deployment system that uses cryptographic hashes to compute unique paths for components⁶ that are stored in a read-only directory: the Nix store, at /nix/store/<hash>-<name>. +This provides several benefits, including concurrent installation of multiple versions of a package, atomic upgrades, and multiple user environments.

Nix uses a declarative domain-specific language (DSL), also called Nix, to build and configure software. +The snippet used to deploy the DNS server is in fact a Nix expression. +This example doesn't demonstrate it, but Nix is Turing complete. +Nix does not, however, have a type system.

We used the DSL to write derivations for software that describe how to build said software with input components and a build script. +This Nix expression is then 'instantiated' to create 'store derivations' (.drv files), which is the low-level representation of how to build a single component. +This store derivation is 'realised' into a built artefact, hereafter referred to as 'building.'

Possibly the simplest Nix derivation uses bash to create a single file containing Hello, World!:

{ pkgs ? import <nixpkgs> {  } }:
+
+builtins.derivation {
+  name = "hello";
+  system = builtins.currentSystem;
+  builder = "${nixpkgs.bash}/bin/bash";
+  args = [ "-c" ''echo "Hello, World!" > $out'' ];
+}

Note that derivation is a function that we're calling with one argument, which is a set of attributes.

We can instantiate this Nix derivation to create a store derivation:

$ nix-instantiate default.nix
+/nix/store/5d4il3h1q4cw08l6fnk4j04a19dsv71k-hello.drv
+$ nix show-derivation /nix/store/5d4il3h1q4cw08l6fnk4j04a19dsv71k-hello.drv
+{
+  "/nix/store/5d4il3h1q4cw08l6fnk4j04a19dsv71k-hello.drv": {
+    "outputs": {
+      "out": {
+        "path": "/nix/store/4v1dx6qaamakjy5jzii6lcmfiks57mhl-hello"
+      }
+    },
+    "inputSrcs": [],
+    "inputDrvs": {
+      "/nix/store/mnyhjzyk43raa3f44pn77aif738prd2m-bash-5.1-p16.drv": [
+        "out"
+      ]
+    },
+    "system": "x86_64-linux",
+    "builder": "/nix/store/2r9n7fz1rxq088j6mi5s7izxdria6d5f-bash-5.1-p16/bin/bash",
+    "args": [ "-c", "echo \"Hello, World!\" > $out" ],
+    "env": {
+      "builder": "/nix/store/2r9n7fz1rxq088j6mi5s7izxdria6d5f-bash-5.1-p16/bin/bash",
+      "name": "hello",
+      "out": "/nix/store/4v1dx6qaamakjy5jzii6lcmfiks57mhl-hello",
+      "system": "x86_64-linux"
+    }
+  }
+}

And build the store derivation:

$ nix-store --realise /nix/store/5d4il3h1q4cw08l6fnk4j04a19dsv71k-hello.drv
+/nix/store/4v1dx6qaamakjy5jzii6lcmfiks57mhl-hello
+$ cat /nix/store/4v1dx6qaamakjy5jzii6lcmfiks57mhl-hello
+Hello, World!

Most Nix tooling does these two steps together:

nix-build default.nix
+this derivation will be built:
+  /nix/store/q5hg3vqby8a9c8pchhjal3la9n7g1m0z-hello.drv
+building '/nix/store/q5hg3vqby8a9c8pchhjal3la9n7g1m0z-hello.drv'...
+/nix/store/zyrki2hd49am36jwcyjh3xvxvn5j5wml-hello

Nix realisations (hereafter referred to as 'builds') are done in isolation to ensure reproducibility. +Projects often rely on interacting with package managers to make sure all dependencies are available and may implicitly rely on system configuration at build time. +To prevent this, every Nix derivation is built in isolation (without network access or access to the global file system) with only other Nix derivations as inputs.

+
The name Nix is derived from the Dutch word niks, meaning nothing; build actions do not see anything that has not been explicitly declared as an input.
+

Nixpkgs

You may have noticed a reference to nixpkgs in the above derivation. +As every input to a Nix derivation also has to be a Nix derivation, one can imagine the tedium involved in creating a Nix derivation for every dependency of your project. +However, Nixpkgs⁷ is a large repository of software packaged in Nix, where a package is a Nix derivation. +We can use packages from Nixpkgs as inputs to a Nix derivation, as we've done with bash.

There is also a command line package manager installing packages from Nixpkgs, which is why people often refer to Nix as a package manager. +While Nix, and therefore Nix package management, is primarily source-based (since derivations describe how to build software from source), binary deployment is an optimisation of this. +Since packages are built in isolation and entirely determined by their inputs, binaries can be transparently deployed by downloading them from a remote server instead of building the derivation locally.

+ + + + +

Visualisation of Nixpkgs⁸

NixOS

NixOS⁹ is a Linux distribution built with Nix from a modular, purely functional specification. +It has no traditional filesystem hierarchy (FSH), like /bin, /lib, /usr, but instead stores all components in /nix/store. +The system configuration is managed by Nix and configured with Nix expressions. +NixOS modules are Nix files containing chunks of system configuration that can be composed to build a full NixOS system¹⁰. +While many NixOS modules are provided in the Nixpkgs repository, they can also be written by an individual user. +For example, the expression used to deploy a DNS server is a NixOS module. +Together these modules form the configuration which builds the Linux system as a Nix derivation.

NixOS minimises global mutable state that -- without knowing it -- you might rely on being set up in a certain way. +For example, you might follow instructions to run a series of shell commands and edit some files to get a piece of software working. +You may subsequently be unable to reproduce the result because you've forgotten some intricacy or are now using a different version of the software. +Nix forces you to encode this in a reproducible way, which is extremely useful for replicating software configurations and deployments, aiming to solve the 'It works on my machine' problem. +Docker is often used to fix this configuration problem, but Nix aims to be more reproducible. +This can be frustrating at times because it can make it harder to get a project off the ground, but the benefits often outweigh the downsides.

Nix uses pointers (implemented as symlinks) to system dependencies, which are Nix derivations for programs or pieces of configuration files. +This means NixOS supports atomic upgrades, as the pointers to the new packages are only updated when the install succeeds; the old versions can be kept until garbage collection. +This also allows NixOS to trivially supports rollbacks to previous system configurations, as the pointers can be restored to their previous state. +Every new system configuration creates a GRUB entry, so you can boot previous systems even from your UEFI/BIOS. +Finally, NixOS also supports partial upgrades: while Nixpkgs also has one global coherent package set, one can use multiple instances of Nixpkgs (i.e., channels) at once, as this Nix store allows multiple versions of a dependency to be stored.

To summarise the parts of the Nix ecosystem that we've discussed:

+ +

Flakes

We also use Nix flakes for this project. +Without going into too much depth, they enable hermetic evaluation of Nix expressions and provide a standard way to compose Nix projects. +With flakes, instead of using a Nixpkgs repository version from a 'channel'¹¹, we pin Nixpkgs as an input to every Nix flake, be it a project build with Nix or a NixOS system. +Integrated with flakes, there is also a new nix command aimed at improving the Nix UI. +You can read more detail about flakes in a series of blog posts by Eelco Dolstra on the topic¹².

Deploying Unikernels

Now that we understand what Nix and Mirage are, and we've motivated the desire to deploy Mirage unikernels on a NixOS machine, what's stopping us from doing just that? +To support deploying a Mirage unikernel, like for a DNS server, we need to write a NixOS module for it.

A paired-down¹³ version of the bind NixOS module, the module used in our Nix expression for deploying a DNS server on NixOS (§), is:

{ config, lib, pkgs, ... }:
+
+with lib;
+
+{
+  options = {
+    services.bind = {
+      enable = mkEnableOption "BIND domain name server";
+      
+      zones = mkOption {
+        ...
+      };
+    };
+  };
+
+  config = mkIf cfg.enable {
+    systemd.services.bind = {
+      description = "BIND Domain Name Server";
+      after = [ "network.target" ];
+      wantedBy = [ "multi-user.target" ];
+
+      serviceConfig = {
+        ExecStart = "${pkgs.bind.out}/sbin/named";
+      };
+    };
+  };
+}

Notice the reference to pkgs.bind. +This is the Nixpkgs repository Nix derivation for the bind package. +Recall that every input to a Nix derivation is itself a Nix derivation (§); in order to use a package in a Nix expression -- i.e., a NixOS module -- we need to build said package with Nix. +Once we build a Mirage unikernel with Nix, we can write a NixOS module to deploy it.

Building Unikernels

Mirage uses the package manager for OCaml called opam¹⁴. +Dependencies in opam, as is common in programming language package managers, have a file which -- among other metadata, build/install scripts -- specifies dependencies and their version constraints. +For example¹⁵

...
+depends: [
+  "arp" { ?monorepo & >= "3.0.0" & < "4.0.0" }
+  "ethernet" { ?monorepo & >= "3.0.0" & < "4.0.0" }
+  "lwt" { ?monorepo }
+  "mirage" { build & >= "4.2.0" & < "4.3.0" }
+  "mirage-bootvar-solo5" { ?monorepo & >= "0.6.0" & < "0.7.0" }
+  "mirage-clock-solo5" { ?monorepo & >= "4.2.0" & < "5.0.0" }
+  "mirage-crypto-rng-mirage" { ?monorepo & >= "0.8.0" & < "0.11.0" }
+  "mirage-logs" { ?monorepo & >= "1.2.0" & < "2.0.0" }
+  "mirage-net-solo5" { ?monorepo & >= "0.8.0" & < "0.9.0" }
+  "mirage-random" { ?monorepo & >= "3.0.0" & < "4.0.0" }
+  "mirage-runtime" { ?monorepo & >= "4.2.0" & < "4.3.0" }
+  "mirage-solo5" { ?monorepo & >= "0.9.0" & < "0.10.0" }
+  "mirage-time" { ?monorepo }
+  "mirageio" { ?monorepo }
+  "ocaml" { build & >= "4.08.0" }
+  "ocaml-solo5" { build & >= "0.8.1" & < "0.9.0" }
+  "opam-monorepo" { build & >= "0.3.2" }
+  "tcpip" { ?monorepo & >= "7.0.0" & < "8.0.0" }
+  "yaml" { ?monorepo & build }
+]
+...

Each of these dependencies will have its own dependencies with their own version constraints. +As we can only link one dependency into the resulting program, we need to solve a set of dependency versions that satisfies these constraints. +This is not an easy problem. +In fact, it's NP-complete ¹⁶. +Opam uses the Zero Install¹⁷ SAT solver for dependency resolution.

Nixpkgs has a large number of OCaml packages¹⁸, which we could provide as build inputs to a Nix derivation. +However, Nixpkgs has one global coherent set of package versions¹⁹. +The support for installing multiple versions of a package concurrently comes from the fact that they are stored at a unique path and can be referenced separately, or symlinked, where required. +So different projects or users that use a different version of Nixpkgs won't conflict, but Nix does not do any dependency version resolution -- everything is pinned. +This is a problem for opam projects with version constraints that can't be satisfied with a static instance of Nixpkgs.

Luckily, a project from Tweag already exists (opam-nix) to deal with this²⁰. +This project uses the opam dependency versions solver inside a Nix derivation, and then creates derivations from the resulting dependency versions.

This still doesn't support building our Mirage unikernels, though. +Unikernels quite often need to be cross-compiled: compiled to run on a platform other than the one they're being built on. +A common target, Solo5²¹, is a sandboxed execution environment for unikernels. +It acts as a minimal shim layer to interface between unikernels and different hypervisor backends. +Solo5 uses a different glibc which requires cross-compilation. +Mirage 4²² supports cross compilation with toolchains in the Dune build system²³. +This uses a host compiler installed in an opam switch (a virtual environment) as normal, as well as a target compiler²⁴. +But the cross-compilation context of packages is only known at build time, as some metaprogramming modules may require preprocessing with the host compiler. +To ensure that the right compilation context is used, we have to provide Dune with all our sources' dependencies. +A tool called opam-monorepo was created to do just that²⁵.

We extended the opam-nix project to support the opam-monorepo workflow with this pull request: github.com/tweag/opam-nix/pull/18. +This is very low-level support for building Mirage unikernels with Nix, however. +In order to provide a better user experience, we also created the Hillinar Nix flake: github.com/RyanGibb/hillingar. +This wraps the Mirage tooling and opam-nix function calls so that a simple high-level flake can be dropped into a Mirage project to support building it with Nix. +To add Nix build support to a unikernel, simply:

# create a flake from hillingar's default template
+$ nix flake new . -t github:/RyanGibb/hillingar
+# substitute the name of the unikernel you're building
+$ sed -i 's/throw "Put the unikernel name here"/"<unikernel-name>"/g' flake.nix
+# build the unikernel with Nix for a particular target
+$ nix build .#<target>

For example, see the flake for building the Mirage website as a unikernel with Nix: github.com/RyanGibb/mirage-www/blob/master/flake.nix.

Evaluation

Hillingar's primary limitations are (1) complex integration is required with the OCaml ecosystem to solve dependency version constraints using opam-nix, and (2) that cross-compilation requires cloning all sources locally with opam-monorepo (§). +Another issue that proved an annoyance during this project is the Nix DSL's dynamic typing. +When writing simple derivations this often isn't a problem, but when writing complicated logic, it quickly gets in the way of productivity. +The runtime errors produced can be very hard to parse. +Thankfully there is work towards creating a typed language for the Nix deployment system, such as Nickel²⁶. +However gradual typing is hard, and Nickel still isn't ready for real-world use despite being open-sourced (in a week as of writing this) for two years.

A glaring omission is that despite it being the primary motivation, we haven't actually written a NixOS module for deploying a DNS server as a unikernel. +There are still questions about how to provide zone file data declaratively to the unikernel and manage the runtime of deployed unikernels. +One option to do the latter is Albatross²⁷, which has recently had support for building with Nix added²⁸. +Albatross aims to provision resources for unikernels such as network access, share resources for unikernels between users, and monitor unikernels with a Unix daemon. +Using Albatross to manage some of the inherent imperative processes behind unikernels, as well as share access to resources for unikernels for other users on a NixOS system, could simplify the creation and improve the functionality of a NixOS module for a unikernel.

There also exists related work in the reproducible building of Mirage unikernels. +Specifically, improving the reproducibility of opam packages (as Mirage unikernels are opam packages themselves)²⁹. +Hillingar differs in that it only uses opam for version resolution, instead using Nix to provide dependencies, which provides reproducibility with pinned Nix derivation inputs and builds in isolation by default.

Conclusion

To summarise, this project was motivated (§) by deploying unikernels on NixOS (§). +Towards this end, we added support for building MirageOS unikernels with Nix: +we extended opam-nix to support the opam-monorepo workflow and created the Hillingar project to provide a usable Nix interface (§).

While only the first was the primary motivation, the benefits of building unikernels with Nix are:

Reproducible and low-config unikernel deployment using NixOS modules is enabled.
Nix allows reproducible builds pinning system dependencies and composing multiple language environments. For example, the OCaml package conf-gmp is a 'virtual package' that relies on a system installation of the C/Assembly library gmp (The GNU Multiple Precision Arithmetic Library). Nix easily allows us to depend on this package in a reproducible way.
We can use Nix to support building on different systems (§).

To conclude, while NixOS and MirageOS take fundamentally very different approaches, they're both trying to bring some kind of functional programming paradigm to operating systems. +NixOS does this in a top-down manner, trying to tame Unix with functional principles like laziness and immutability³⁰; whereas, MirageOS does this by throwing Unix out the window and rebuilding the world from scratch in a very much bottom-up approach. +Despite these two projects having different motivations and goals, Hillingar aims to get the best from both worlds by marrying the two.

To dive deeper, please see a more detailed article on my personal blog.

If you have a unikernel, consider trying to build it with Hillingar, and please report any problems at github.com/RyanGibb/hillingar/issues!

ISC bind has many CVE's ↩
mirage.io ↩
Credits to Takayuki Imada↩
Barring the use of foreign function interfaces (FFIs).↩
As 'nix' means snow in Latin. Credits to Tim Cuthbertson.↩
NB: we will use component, dependency, and package somewhat interchangeably in this blog post, as they all fundamentally mean the same thing -- a piece of software.↩
github.com/nixos/nixpkgs ↩
www.tweag.io/blog/2022-09-13-nixpkgs-graph/↩
nixos.org ↩
NixOS manual Chapter 66. Writing NixOS Modules.↩
nixos.org/manual/nix/stable/package-management/channels.html ↩
tweag.io/blog/2020-05-25-flakes ↩
The full module can be found here ↩
opam.ocaml.org ↩
For mirage-www targetting hvt.↩
research.swtch.com/version-sat ↩
0install.net ↩
github.com/NixOS/nixpkgs pkgs/development/ocaml-modules ↩
Bar some exceptional packages that have multiple major versions packaged, like Postgres.↩
github.com/tweag/opam-nix ↩
github.com/Solo5/solo5 ↩
mirage.io/blog/announcing-mirage-40 ↩
dune.build ↩
github.com/mirage/ocaml-solo5 ↩
github.com/tarides/opam-monorepo ↩
www.tweag.io/blog/2020-10-22-nickel-open-sourcing ↩
hannes.robur.coop/Posts/VMM ↩
https://github.com/roburio/albatross/pull/120 ↩
hannes.nqsb.io/Posts/ReproducibleOPAM ↩
tweag.io/blog/2022-07-14-taming-unix-with-nix ↩

OCaml 5 Release Candidate Now Available!

tarides — 2022-12-07T00:00:00-00:00

We're in the home stretch for the full OCaml 5 release. Multicore is almost here! Yesterday its Release Candidate (RC) was announced on the OCaml Discuss, which is the final step before the major release, expected before Christmas.

To learn more about the exciting features coming with OCaml 5, you can watch KC’s keynote address and check out his speaker slide deck as well. As always, feel free to contact us for more information about using OCaml and for support on your OCaml projects.

The OCaml community has worked tirelessly to release 5.0 before the end of the year, with a lot of time spent on creating a smooth transition for OCaml users. There should be just enough time for you to try out OCaml 5 for a fun holiday project or the Advent of Code.

Your reports resulted in these bug fixes since the Beta 2 release last week:

11776: Extend environment with functor parameters in strengthen_lazy. (Chris Casinghino and Luke Maurer, review by Gabriel Scherer)
11533 and 11534: follow synonyms again in #show_module_type (this had stopped working in 4.14.0) (Gabriel Scherer, review by Jacques Garrigue, report by Yaron Minsky)

For the full change log, visit the GitHub repo. The source code for the release candidate is available at these addresses:

Please keep those testing reports coming in. We believe this release candidate is ready to go, but we really value testing right up to the last minute to be even more sure. Send us your valuable input! If you find something, please open an issue on GitHub or join the discussion on the Discuss post, where you can also find installation instructions.

OCaml 5 Beta2 Release

tarides — 2022-11-29T00:00:00-00:00

Just about a month after the OCaml 5 Beta release, the OCaml 5 Beta2 version has been released, taking us one step closer to the full OCaml 5 with Multicore release later this year. The OCaml community's collaboration is coming to fruition! Although we're not quite ready for the RC1 (Release Candidate) version, several things have been added and improved with Beta2.

To learn more about the exciting things coming with OCaml 5, please watch KC Sivaramakrishnan’s keynote address and check out his speaker slide deck as well. As always, feel free to contact us for more information about using OCaml and for support on your OCaml projects.

Here's a partial list of improvements/fixes with issue numbers:

#11631 - fix an assertion dealing with a segfault found by the Multicore test suite
#11662, #11673 - memory leak affecting dynlink with frame descriptor tables (reported by Frama-C devs)
#11704, #11669 - segfault with effects fixed, having been tracked down to the refactoring of Effect.Unhandled
#11701 - fix spurious .dSYM files and directories being created on macOS
#11671 - bug in top_heap_words statistics accounting fix
#11670 - macOS fix when creating empty archives
#11097 - NetBSD fixes, including ARM64 support
#11194, #11609 - fixes a regression from 4.14
#11622 - fixes a regression in error messages since 4.10
#11725 - remove caml_alloc_N
#11661 - erroneous -force-tmc option removed
#11367, #11652 - Windows clean-ups
#11611 - fix --disable-instrumented-runtime
#11639 - configuration bookkeeping (ensure system, etc., always set)
#11632 - minor bookkeeping bug fix
#11559, #11649, #11640, #11301, #11705 - docs updates

In short, we're continuing to stabilise the release. We're also dealing with reports coming from the wonderful testing that's been going on, especially Multicore tests and the Frama-C report. Keep those testing reports and feedback coming by opening an issue on the GitHub repo or chiming in through the OCaml Discuss forum post.

Thanks to the hard work by all engineers working to make OCaml even better than before. It's a beautiful sight to watch brilliant developers come together on an open-source project like OCaml, and Tarides is proud to be part of this ever-growing community.

Solve the 2022 Advent of Code Puzzles with OCaml

tarides — 2022-11-24T00:00:00-00:00

Too many programmers only know OCaml through a functional programming language overview course at university. They erroneously believe OCaml is used primarily in academia rather than in the real world. Not only is OCaml already used in several prominent businesses, it can also be used for fun projects, like the upcoming Advent of Code.

What is Advent of Code?

Advent of Code is an annual online Advent calendar produced specifically for programmers. It publishes a series of daily puzzles, revealing a new puzzle every day from 1 December to 25 December. That's 25 days of coding challenges! These puzzles can be solved in the language of your choice.

Every year they have new puzzles and anyone can participate, whether you're new to programming or a veteran. Advent of Code has been running since 2015 and has attracted a large community of developers around the world!

It's not only a fun way to learn new languages, but it also provides a great way to meet new people with the same interests and exchange ideas.

OCaml 5 is set for release later this year, so it's a perfect time to learn and practice OCaml with Advent of Code!

5 Reasons to Learn OCaml

There are so many misconceptions around OCaml that it's hard to know where to start. Basically, OCaml can do what Python, C++, Java, or any other major programming language can do. Here are a few specific reasons why OCaml should be the next language you learn:

1. Secure-By-Design

Cyber attacks have become more commonplace and sophisticated, so security has become a top priority in software development. With the proliferation of cloud services and Internet-connected devices, software must be secure-by-design to prevent malicious actors from taking advantage of any bugs or loopholes. Creating software with a secure-by-design language like OCaml helps meet this goal.

OCaml has built-in features and design patterns, like type and memory safety, that make it secure-by-design.

2. Performant Garbage Collector

Contrary to popular belief, OCaml's garbage collector (GC) doesn't slow things down because it's incremental, which can help avoid the problems of manual memory management in large or long-running programs.

OCaml's GC has to run periodically, but it can do so in small incremental steps. Although allocations that trigger a GC are longer than a malloc call (used in C), most of them are almost immediate because allocating from the minor heap is as cheap as allocating on the stack.

This incremental GC avoids the problems normally associated with garbage collection, like tying up memory and slowing down the process.

3. Confidence in the Code Through Type Checking

OCaml's compile-time type checking eliminates many potential runtime errors, and its strong type inference eliminates several redundant type annotations. Often, a programming language has either type safety or type inference, but with OCaml you get both!

4. Multicore!

With the release of OCaml 5 later this year comes Multicore support! Multicore is an extension of OCaml with native support for Shared-Memory Parallelism through domains and Concurrency through algebraic effects. The ability to run OCaml on multiple cores will make it even faster than before.

5. Extensive Tools & Libraries

OCaml has some great tools and helpful libraries, like MirageOS, Irmin, and so many others as reported by seasoned OCaml programmers in this Discuss thread.

OCaml platform tools include the Dune build system, opam package manager, Merlin IDE, and the odoc documentation generator.

Pro Tip: Learn OCaml Basics First

Before you start solving puzzles, learn the basics to ensure you understand the most common data structures and algorithms. It's important to know how to print to the console, read and write files, and parse text.

It’s also a good idea to read about common problem-solving approaches. For example, checking whether a solution is correct is a crucial part of the solving process. This will help you understand the challenges better, and you can save yourself a lot of time and frustration.

Get Up & Running with OCaml today through the tutorials on OCaml.org. Also consider joining the OCaml Community Forum Discuss. They're very welcoming of those new to OCaml as well as experienced OCaml programmers, and members will quickly answer any questions you have while you learn.

Conclusion

The Advent of Code is a great way to practise your problem-solving skills in a new programming language during the holidays.

Give OCaml a try this holiday season. You won't regret it!

Learn more about the forthcoming OCaml 5 through KC Shivaramakrishnan's Keynote Address and speaker deck. Stay tuned to this blog for release updates and a series of posts about why you should consider OCaml as your next language.

Six Surprising Reasons the OCaml Programming Language is Good for Business

tarides — 2022-11-22T00:00:00-00:00

Functional programming languages have been around since the 1950's, when the first high-level languages were used to program early computers. Examples of functional programming languages include OCaml, Erlang, Clojure, Haskell, Scala, and Common Lisp. Choosing the right programming language is critical to the long-term success and stability of your products, services, and operations. With strong academic roots and years of iteration and innovation, functional programming languages can offer a real competitive edge to businesses. This article explains some of the lesser-known benefits of OCaml from a business perspective.

Why Functional Programming?

The strengths of functional programming are becoming increasingly well-known. Functional programming lets programmers write programs in a declarative, logical, and mathematical style. This makes it easier for the developer to express their intent in a declarative style, where the code closely matches the specification. A specification is a mathematical description of what a program does, and programs that match their specification are proven to follow that description, which makes them predictable and safe. Furthermore, with features that limit the mutation of data and side effects, functional programs tend to have fewer bugs and vulnerabilities, remain easy to develop and maintain, and last longer overall.

Nowadays, widely-used imperative programming languages such as Python, Rust, and Java support programming in a functional style and use functional programming features to take advantage of its strengths. Features like rich type systems, pattern matching, and lambda expressions are becoming more mainstream, illustrating their usefulness.

What Makes OCaml Unique?

OCaml combines a strong foundation in functional programming with some select imperative and object-oriented programming features, allowing the user to choose the best approach for the task at hand. This is part of what makes OCaml such a great general-purpose programming language; it combines the strengths of several programming styles and offers the developer a full software development toolkit.

OCaml provides a unique balance of performance, security, and reliability. It combines features like a garbage collector, static type-checking, type-driven development, first-class functions, and pattern matching (features that work well together). Together they result in a language known for minimising errors, debugging easily, automatically managing memory, preventing structural errors in data, and providing a user-friendly developer environment.

When OCaml 5 is released later this year, the language will get a significant upgrade, introducing support for shared-memory parallelism and native support for simple concurrent programming. Running programs on multiple cores will allow developers to considerably reduce the runtime of their projects by executing code in parallel. The quality-of-life updates to concurrent programming will make it easier for developers to write high-performance concurrent code.

So what are some of OCaml’s greatest strengths? Here are the key reasons why businesses use OCaml to solve complex, critical, and time-sensitive problems.

1. OCaml Is Trusted By Several Prominent Companies

Think of OCaml as an academic language? Think again! Owing to its reputation for reliability, safety, and performance, many businesses use it to solve real-world problems. Talented software engineers all over the world create robust, maintainable, energy-efficient, and fast solutions in OCaml for high-pressure environments where a single mistake can cost millions.

Docker is making life easier for developers by providing them with a state-of-the-art integrated development pipeline that consolidates application components, all available conveniently on your desktop. Docker has over fifteen million registered users worldwide and uses VPNKit, which is written in OCaml, in its Docker Desktop app to keep user networks secure.

Meta is a multiplatform company that uses tech to bring people together and build unique communities online. The social media giant uses OCaml in major parts of its infrastructure, such as the compiler and typechecker for its programming language Hack. Other Meta tooling that uses OCaml includes Infer, Flow, and the now retired Pfff.

Jane Street is a quantitative trading firm that uses OCaml as their core solution for their research tools, trading systems, and accounting systems. Processing billions of dollars each day, Jane Street relies on OCaml to ensure it is done securely and quickly. Notably, nearly a million lines of their code is open source, and they work closely with the OCaml community to develop the language and its tools.

Other companies that use OCaml include Ahrefs, an all-in-one SEO tool; Nitrokey, a world-leading provider of open-source security hardware; and Hyper, who use OCaml to provide their customers with a unified data platform to manage large infrastructure. There are also two blockchains written in OCaml: Tezos and Mina. When it comes to the modern landscape of software development, it is safe to say OCaml has an ever-growing part to play.

The Takeaway : Several top companies are already using OCaml, so there is a strong tradition of successfully using OCaml on an industrial level.

2. OCaml Has a Growing and Thriving Community

The open-source community surrounding OCaml is diverse and flourishing. It congregates in several places online, like the Discuss forum, GitHub repo, and Reddit community. Thanks to its increasing popularity, more and more tutorials and documentation are becoming available online. Learning new languages with the help of online material is the trend nowadays, and OCaml welcomes all developers with open arms. In fact, there is a great book for learning OCaml entirely available online called Real World OCaml.

Over thirty universities currently teach OCaml, including the University of Cambridge, Paris-Diderot University, the Indian Institute of Technology Madras, Harvard, and Cornell University. Cornell University has a great textbook on OCaml. Students who learn OCaml often end up becoming part of its open-source community, contributing to projects and launching initiatives of their own. Companies that use OCaml to provide their customers with great products also contribute to the OCaml community, as do academics, researchers, and hobbyists. All entry-points contribute to the development of the language, and users end up interacting with each other across these categories, each bringing their own unique perspective.

The Takeaway : The community surrounding OCaml is vibrant and thriving, allowing you to invest your time and energy in OCaml knowing it’s here to stay.

3. OCaml Offers Powerful Tools and Plenty of Support

OCaml is not only an industrial-strength programming language, but it also provides industry-ready tooling and ecosystem support. The OCaml Platform is a curated set of tools that have broad community support. It includes all the tools you'd expect from an industrial-strength programming language, including a build system, package manager, editor support, and documentation generator. The OCaml Platform tells you whether one of these tools is active, under incubation, or deprecated. The OCaml Platform ensures that developers not only have an excellent language at hand but also have the tools to productively develop software with that language. Furthermore, the OCaml website has thorough resources on everything OCaml, including a comprehensive guide to setting up OCaml on your computer.

The OCaml compiler is regularly updated and has a dedicated team focused on innovation, improving new features, and keeping everything bug-free. If you have a problem or need help, the response time across the various OCaml forums is very quick and supportive of new learners. Recently, the aforementioned great and comprehensive book Real World OCaml has been updated for its 2nd edition. This edition is available to download online as a free PDF to make learning OCaml even more accessible.

If you want to create something new in OCaml, or need help building something, there are several companies as well as hobbyist groups to consult. You can find like-minded people to discuss your ideas with on the OCaml forum Discuss. Tarides is one of the companies that regularly work on OCaml, so you can send us a message if you’d like help with a project.

The Takeaway : OCaml offers several up-to-date tools supported by an active group of contributors and maintainers. Using OCaml means getting help fast and having a complete developer environment with everything you need.

4. OCaml is Secure-By-Design

In today’s connected world, keeping yourself and your data safe online is not just a matter of convenience, but it's also a crucial task that can have serious personal and professional repercussions. Luckily, the OCaml programming language is built in a way that promotes safety, including features and design patterns that make it secure-by-design. It's easy to integrate formally verified code with OCaml programs, which makes OCaml more secure. Some formally verified libraries available in OCaml include Microsoft's HACL* and MIT's Fiat.

Secure-by-design is a known programming term which means that a language is constructed in a way that fundamentally promotes security and minimises vulnerabilities. A language that is secure-by-design makes it impossible to introduce a large class of security vulnerabilities into programs written using that language. A great example of how secure-by-design principles are implemented in OCaml is its type and memory safety, which prevents the most frequent kinds of attacks and crashes from ever happening.

Memory-safety attacks are extremely common, with approximately 70% of zero-day attacks being memory-safety attacks. OCaml is memory safe because it doesn’t allow a pointer (the designator of the information being written into memory) to enter information into an unauthorised memory block. As a result, you can’t make a program crash with OCaml by manipulating where it writes code into memory, as OCaml simply does’t allow this to happen. This prevents programs crashing due to memory exploits, including buffer overflows, where memory is ‘tricked’ into writing more than the block ‘allows.’

OCaml is also statically-typed and type-safe, meaning that it detects errors at compile time and completely stops programs with defects from running, as well as limits what type of operations can be performed on which kinds of data. Both work to remove bugs and errors from the code, making programs written in OCaml more reliable and consistent.

The Takeaway : Cybersecurity is an increasing area of concern, and OCaml has several built-in features that help make it secure-by-design. It’s an excellent choice for projects where security is paramount.

5. OCaml is Big on Performance and Developer Productivity

OCaml is hailed for striking a balance between a large number of advanced features and performance. For example, it has a very efficient compiler that’s divided into two parts: a bytecode compiler and a native compiler. The bytecode compiler is very quick and generates small, portable executables. The native code compiler produces highly-efficient machine code.

Since OCaml also allows for some uses of imperative and object-oriented programming features, it’s possible to use them in places where they can help with performance. This flexibility of programming paradigms is another way that OCaml helps programmers increase the speed and efficiency of the code they write.

Type inference allows the language to infer what type is being used, removing the need for the developer to annotate every single variable in their code. This makes developing in OCaml faster than many other languages that lack type inference. OCaml also allows the developer to write complex algorithms without introducing bugs. The developer can easily optimise algorithms for greater speed without compromising performance and security. Furthermore, the presence and use of algebraic data types, higher order functions, and immutable data all make manipulating large and complex data structures much easier and faster.

It is also worth noting that OCaml offers several strong methods for debugging its programs. From the fast, interactive, REPL to the powerful symbolic replay debugger, OCaml lets you eliminate bugs at compile time and avoid them at runtime. This, in combination with how effective the debugging programs are, makes OCaml an easy language to debug. This saves developer time and increases the productivity and speed of programming.

The Takeaway : The OCaml language is strong on performance and has a lot of features that make the code run fast, while also making the development process more efficient.

6. OCaml is Multicore!

With the imminent release of OCaml 5, the language will support the use of multiple cores and have enhanced infrastructure in place for concurrent programming. Both bring significant performance boosts, allowing users to increase the speed of their programs.

The new I/O library Eio can serve more than one million requests per second, outperforming Go’s net/http and closely matching Rust’s hyper. Writing concurrent code will also be much easier in OCaml 5, just like writing regular OCaml. The ‘function colouring problem,’ whereby concurrent and non-concurrent code are incompatible, will also be a thing of the past after the new release. With OCaml 5, both kinds of code can coexist with minimal intervention on the part of the programmer.

Multicore or parallel programming increases the efficiency of a program by several orders of magnitude. By letting the computer use more than one core to execute the code, the program can do several things simultaneously rather than consecutively. For complex tasks that take a long time, Multicore revolutionises their applicability, making them more realistic and time-efficient alternatives.

You can look forward to more posts on the Tarides blog about OCaml 5, the technology behind it, and its use cases.

The Takeaway : OCaml 5 is coming, and with it, Multicore. OCaml will become even more powerful, both for parallel programming and concurrent programming, allowing users to significantly boost the speed of their projects.

Conclusion

Choosing the perfect programming language for you is an important but difficult task. It needs to be powerful and guarantee performance while simultaneously offering strong security features. Developers are also going to need state-of-the-art tools alongside responsive help and support. OCaml is a good candidate that offers all of the above, making it great for businesses looking for a versatile and robust programming language.

Combining the power of functional programming, Multicore, and open source, OCaml offers a potent mix of strong features and an engaged community. For more information about OCaml, you can visit the OCaml Website or contact Tarides to see how we can make OCaml work for you.

OCaml 5 at Open Source India 2022

tarides — 2022-11-16T00:00:00-00:00

Open Source India 2022

With OCaml 5 just around the corner, it's been a really exciting year to attend conferences all over the world. Just recently, I presented some highlights of the OCaml 5 update, building on KC Sivaramakrishnan's great keynote address at the 19th annual Open Source India 2022 conference. Since Tarides was invited to participate, I gave a talk on OCaml 5: Language, Platform, and Ecosystem by starting with OCaml's history and ending with a Multicore OCaml matrix implementation running on 120 cores!

Open Source India was held on the 29-30th September 2022 as a physical event at the NIMHANS Convention Centre, Bengaluru, India. It was organised by the Open Source For You magazine team in India, with the help of community and industry participation. The conference ran along multiple parallel "tracks" - FOSS for Everyone, Developers, CXO Summit, DevOps, AI & ML, Data Management, and IT Infrastructure. My talk was part of the Developers track on the second day.

Day I

The conference had many exhibits, and I interacted with a number of participants at the booths. MOSIP is an open source platform for national foundational identities. Some Governments implement a digital identity system for its citizens, and MOSIP provides a robust, scalable, open source platform for governance. While they currently use PostgreSQL as their database backend, it would be useful to re-model their backend to use Irmin as the data store for security reasons.

Another Business-to-Consumer (B2C), open-source software application was Chatwoot, a customer engagement and support platform that also uses PostgreSQL. It would be an interesting data modeling or solution architect project to implement Irmin support for their chat application. The Compossible Umwelten company are working on wearable computing using ARM processors, and they were interested in exploring using OCaml, instead of C, for their customer products.

Post-lunch, I attended a talk on Open Source at AWS by Suman Debnath, Principal Developer Advocate, Data Engineering and Analytics at Amazon Web Services. We had the opportunity to discuss the possibility of providing the OCaml Platform and Products available through Amazon directly to end users.

In the afternoon, I took the time to attend the AI & ML track. The Adopting MLSecOps talk was presented by Dibya Prakash, CTO and Principal Consultant at Neural Hub, and he introduced me to MLSecOps and best practices in the industry. This was followed by a talk on Time Series Analysis: Anticipating Future with Darts by Binitha MT and Subhankar Adak from Dell. It was an interesting first day at the conference with useful discussions on technology and real world experiences.

Day II

In the morning, I spent some time at the speaker's lounge reviewing the slides for OCaml 5, as well as setting up the demo for the Multicore OCaml code examples. I also had the chance to meet my colleague, Puneeth Chaganti, who works remotely from Bengaluru.

+ + + + +

That afternoon, I gave my talk on OCaml 5: Language, Platform, and Ecosystem. After reviewing OCaml's history, I discussed the recent Multicore OCaml merge, then delved into the syntax of the OCaml 5 language: basic types, operations, control structures, data structures, user types, functions, recursion, and I/O. Following this, I showed examples of domainslib and Eio before demonstrating the impressive Multicore OCaml matrix implementation running on 120 cores!

Additionally, I presented the various platform tools available in the OCaml community, including the OCaml package manager (opam), the Dune build system, odoc, OCaml-LSP, Merlin, and MDX. I also introduced the following ecosystem projects: Sandmark benchmarking suite, Tezos blockchain, Irmin database, MirageOS library operating system, Dream web framework, and OCaml Scientific Computing project. I finished my talk with some useful references for OCaml. To my delight, the participants were curious to learn more!

The conference gave me a great opportunity to reach out to developers and make them aware of the current state of OCaml. It was good to share the platform and ecosystem projects with them so that they can get started with their contributions. I look forward to participating in more conferences and promoting the use of OCaml!

Presenting on Algebraic Effects at FP-SYD

tarides — 2022-11-15T00:00:00-00:00

At ICFP this year, KC Sivaramakrishnan gave two talks that put OCaml 5 in the spotlight: his keynote, “Retrofitting Concurrency - Lessons from the Engine Room,” and the opening presentation of the OCaml workshop, “OCaml 5.0 - Concurrent and Parallel Programming.” Effect Handlers feature heavily, as they are the foundations on which concurrency primitives were added to OCaml. Since I knew very little about effects in this context, I asked KC for some pointers on where to start with learning about them. He pointed me to the Koka programming language, encouraging me to set it up, play with it, and see how its type systems work with effects and effect handlers.

Following the usual tradition of learning something by committing to give a talk on it, I signed up to speak about algebraic effects at my local functional programming meetup, FP-SYD. I figured that I would have a friendly audience that would be forgiving of excessive hand waving! :)

Brain Food

I had an absolute blast learning about algebraic effects! Koka was really easy to install and use. Its rich set of examples and excellent documentation make it work really well as a place to explore the concepts of effect systems.

That got me started, but what really hooked me was discovering Andrej Bauer’s paper, What is Algebraic about Algebraic Effects and Handlers. As a mathematician, I found it to be an accessible way of getting to the topic's theoretical underpinnings (see here for the paper and videos). Next, Matija Pretnar’s tutorial, An Introduction to Algebraic Effects and Handlers, complimented Bauer's work really well.

As I learned more about the topic, I realised just how deep an area this is, and I realised how little I would know about it in four weeks of enthusiastic (but decidedly surface-level) reading. So, I decided to pitch the talk as an overview or a survey of the field. To make things concrete, I would also focus on examples from the effect handling systems of Koka and OCaml 5.

The Talk

October’s FP-SYD was on the 19th. Around 6pm in Sydney’s Central Business District, about twenty people showed up, ate pizza, and settled into general functional programming-related geekery. Haskell is very strongly represented in this community, and it turned out that most of the audience had not spent much time with effect systems, as their tools for working with effects have been Monads, Monad Transformers, and accompanying abstractions.

I enjoyed talking through what I had learned about Algebraic Effects. Since I've worked on Haskell teams, I connected with the community on the various ways computational effects are handled between pure and impure functional languages. It was great to reference Alexis King's recent work that resulted in delimited continuations being added to GHC. The examples from Koka helped give folks a taste for the type systems that include effects and values. I also leaned on the excellent work of KC and the Tarides team, borrowing heavily from the material they have written on effect handlers for the OCaml manual.

The talk certainly delivered, as it got me started on the vast topic of algebraic effects. It also piqued the curiosity of at least a few people in my community, so that’s not nothing! :)

Acknowledgements

I am particularly grateful to KC for getting me curious and giving me material and inspiration to get started. Thanks also to Sudha Parimala for putting together a comprehensive tutorial on parallelism and effects in OCaml and for answering questions and making helpful suggestions. My colleague and friend Tim McGilchrist runs the FP-SYD meetup, so he helped by asking a lot of really good questions, listening to a draft version of the talk, and generally providing support and encouragement. Thanks, Tim! :)

The slides of my talk are available on the FP-SYD repository.

Towards Minimal Disk-Usage for Tezos Bakers

tarides — 2022-11-10T00:00:00-00:00

Over the last few months, Tarides has focused on designing, +prototyping, and integrating a new feature for Tezos bakers: automatic +context pruning for rolling and full nodes. This feature will allow +bakers to run Tezos with minimal disk usage while continuing to enjoy +12x more responsive operations. The first version has been +released with Octez v15. The complete, more optimised context pruning +feature will come with Octez v16. We encourage every Tezos baker to +upgrade and give feedback.

We have implemented context pruning for rolling and full nodes, which +requires ~35GB of disk for storing 6 cycles in the upper layer. In +Octez v15, each subsequent pruning run needs an additional 40GB, but +that space is recovered when the operation finishes. We plan to remove +that extra requirement in Octez v16.

Improve Space Usage with Context Pruning

The Tezos context is a versioned key/value store that associates for +each block a view of its ledger state. The versioning uses concepts +similar to Git. The current implementation is using irmin as +backend and abstracted by the lib_context library.

We have been designing, prototyping, and integrating a new structure +for Irmin storage. It is now reorganised into two +layers: one upper layer that contains the latest cycles of the +blockchain, which are still in use, and a lower layer containing +older, frozen data. A new garbage collection feature (GC) periodically +restructures the Tezos context by removing unused data in the oldest +cycles from the upper layer, where only the data still accessible from +the currently live cycles are preserved. The first version of the GC, +available in Octez-v15, is optimised for rolling and full nodes and +thus does not contain a lower layer. We plan to extend this feature in +Octez-v17 to dramatically improve the archive nodes' performance by +moving the unused data to the lower layer (more on this below).

Garbage collection and subsequent compression of live data improves +disk and kernel cache performance, which enhances overall node +performance. Currently, rolling nodes operators must apply a +manual cleanup process to release space on the disk by discarding +unused data. The manual cleanup is tedious and error-prone. Operators +could discard valuable data, have to stop their baker, or try to devise +semi-automatic procedures and run multiple bakers to avoid +downtime. The GC feature provides rolling nodes operators +a fully automated method to clean up the unused data and guarantees +that only the unused data is discarded, i.e., all currently used data +is preserved.

The GC operation is performed asynchronously with minimal impact on +the Tezos node. In the rolling node's case, a GC'd context uses less +disk space and has a more stable performance throughout, +as the protocol operations (such as executing smart contracts or +computing baking rewards) only need data from the upper layer. As +such, the nodes that benefit from the store's layered structure don't +need to use the manual snapshot export/import—previously necessary when +the disk’s context got too big. In the future, archive nodes’ +performance will improve because only the upper layer is needed to +validate recent blocks. This means archive nodes can bake as reliably +as rolling nodes.

Tezos Storage, in a Nutshell

The Tezos blockchain uses Irmin as the main storage component. Irmin +is a library to design Git-like storage systems. It has many backends, +and one of them is irmin-pack, which is optimised for the Tezos use +case. In the followings, we focus on the main file used to store +object data: the store pack file.

Pack file: Tezos state is serialised as immutable functional objects. +These objects are marshalled in a append-only pack file, one after the +other. An object can contain pointers to the file's earlier (but not +later!) objects. Pointers to an earlier object are typically +represented by the offset (position) of the earlier object in the +pack file. The pack file is append-only: existing objects are +never updated.

+ + + + +

+
An Irmin pack file as a sequence of objects: | obj | obj | obj | ...
+

Commit objects: Some of the objects in the pack file are commit +objects. A commit, together with the objects reachable from that +commit, represents the state associated to a Tezos' block. The +Tezos node only needs the last commit to process new blocks, but +bakers will need a lot more commits to compute baking rewards. +Objects not reachable from these commits can are unreachable or dead +objects.

+ + + + +

+
The data-structure (mental model) representation of the pack file vs. its physical representation.
+

Archive nodes and rolling nodes: There are different types of +Tezos nodes. An archive node stores the complete blockchain +history from the genesis block. Currently, this is over 2 million +blocks. Roughly speaking, a block corresponds to a commit. A +rolling node stores only the last n blocks, where n is chosen +to keep the total disk usage within some bounds. This may be as small +as 5 (or even less) or as large as 40,000 or more. Another type of +node is the "full node," which is between an archive node and a +rolling node.

Rolling nodes, disk space usage: The purpose of the rolling node +is to keep resource usage, particularly disk space, bounded by only +storing the last blocks. However, the current implementation does +not achieve this aim. As rolling nodes execute, the pack file +grows larger and larger, and no old data is discarded. To get around +this problem, node operators periodically export snapshots of the +current blockchain state from the node, delete the old data, +and then import the snapshot state back.

Problem summary: The main problem we want to avoid is Tezos users +having to periodically export and import the blockchain state to +keep the disk usage of the Tezos node bounded. Instead, we want to +perform context pruning via automatic garbage collection of unreachable +objects. Periodically, a commit should be chosen as the GC +root, and objects constructed before the commit that are not +reachable from the commit should be considered dead branches, removed from +the pack store, and the disk space reclaimed. The problem is that +with the current implementation of the pack file, which is just an +ordinary file, it is impossible to "delete" regions corresponding to +dead objects and reclaim the space.

Automatised Garbage Collection Solution

Consider the following pack file, where the GC-commit object has +been selected as the commit root for garbage collection:

+ + + + +

Objects that precede the commit root are either reachable from the +commit (by following object references from it) or not. For the +unreachable objects, we want to reclaim the disk space. For reachable +objects, we need to be able to continue to access them via their +offset in the pack file.

The straightforward solution is to implement the pack file using two +other data structures: the suffix and the prefix. The suffix +file contains the root commit object (GC-commit) and the live +objects represented by all bytes following the offset of GC-commit +in the pack file. The prefix file contains all the objects +reachable from the root commit, indexed by their offset. Note that the +reachable objects appear earlier in the pack file than the root +commit.

+ + + + +

+
The layered structure of the pack file with prefix+suffix as the upper layer.
+

Reading from the pack file is then simulated in an obvious way: if +the offset is for the GC-commit, or later, we read from the suffix +file, and otherwise, we lookup the offset in the prefix and return +the appropriate object. We only access the reachable objects in the +prefix via their offset. We replace the Irmin pack file with +these two data structures. Every time we perform garbage collection +from a given GC-commit, we create the next versions of the prefix +and suffix data-structures and switch from the current version to the next +version by deleting the old prefix and suffix to reclaim +disk space. Creating the next versions of the prefix and suffix +data-structures is potentially expensive. Hence, we implement these steps in a +separate process, the GC worker, with minimal impact on the running +Tezos node.

Caveat: Following Git, a commit will typically reference its +parent commit, which will then reference its parent, and so +on. Clearly, if we used these references to calculate object +reachability, all objects would remain reachable forever. However, +this is not what we want, so when calculating the set of reachable +objects for a given commit, we ignore the references from a commit +to its parent commit.

The `prefix` Data-Structure

The prefix is a persistent data-structure that implements a map from +the offsets in pack file to objects (the marshalled bytes +representing an object). In our scenario, the GC worker creates the +prefix, which is then read-only for the main process. Objects are +never mutated or deleted from the prefix file. In this setting, a +straightforward implementation of an object store suffices: we store +reachable objects in a data file and maintain a persistent (int → int) map from "offset in the original pack file" to "offset in the +prefix file."

Terminology: We introduce the term "virtual offset" for "offset in +the original pack file" and the term "real offset" for "offset in +the prefix file." Thus, the map outlining virtual offset to real +offset is made persistent as the mapping file.

Example: Consider the following, where the pack file contains +reachable objects o1 .. o10, (with virtual offsets v1 .. v10, respectively):

+ + + + +

Note that the objects o1 .. o10 are scattered throughout the +pack file where they appear in ascending order (i.e., v1 < .. < +v10). The prefix file contains the same objects but with different +"real" offsets r1..r10, as now the objects o1 .. o10 appear one +after the other. The mapping needs to contain an entry (v1 → r1) +for object o1 (and similarly for the other objects) to relate the +virtual offset in the pack file with the real offset in the prefix +file.

To read from "virtual offset v3" (say), we use the map to retrieve +the real offset in the prefix file (i.e., r3) and then read the object +data from that position.

Asynchronous Implementation

Tezos Context pruning is performed periodically. We want each round of +context pruning to take place asynchronously with minimal impact +on the main Tezos node. For this reason, when a commit is chosen as +the GC root, we fork a worker process to construct the next prefix +and suffix data structures. When the GC worker terminates, the main process +handles worker termination. It switches from the current +prefix+suffix to the next and continues operation. This +switch takes place almost instantaneously. The hard work is done in +the worker process as depicted next:

+ + + + +

Read-only Tezos nodes: In addition to the main Tezos read/write +node that accesses the pack store, several read-only nodes also +access the pack store (and other Irmin data files) in read-only +mode. These must be synchronised when the switch is made from the +current prefix+suffix to the next prefix+suffix. This +synchronisation makes use of a (new) single control file.

Further Optimisations in the Octez Storage Layer

The context pruning via automatic garbage collection performs well and +within the required constraints. However, it is possible to make +further efficiency improvements. We next describe some potential +optimisations we plan to work on over the next months.

Resource-aware garbage collection:

The GC worker intensively uses disk, memory, and OS resources. For +example, the disk and memory are doubled in size during the +asynchronous execution of the GC worker. We plan to improve on this by +more intelligent use of resources. For example, computing the +reachable objects during the GC involves accessing earlier objects, +using a lot of random-access reads, with unpredictable latency. A more +resource-aware usage of the file system ensures that the objects are +visited (as much as possible) in the order of increased offset on +disk. This takes advantage of the fact that sequential file access is +much quicker and predictable than accessing the file randomly. The work on +context pruning via a resource-aware garbage collection is planned to +be included in Octez v16.

Retaining older objects for archive nodes:

Archive nodes contain the complete blockchain history, starting from +the genesis block. This results in a huge store pack file, many +times larger than the kernel’s page cache. Furthermore, live objects +are distributed throughout this huge file, which makes it difficult +for OS caching to work effectively. As a result, as the store becomes +larger, the archive node becomes slower.

In previous prototypes of the layered store, the design also included a +"lower" layer. For archive nodes, the lower layer contained all the +objects before the most recent GC-commit, whether they were reachable +or not. The lower layer was effectively the full segment of the +pack file before the GC commit root.

One possibility with the new layout introduced by the GC is to retain the +lower layer whilst still sticking with the prefix and mapping files +approach and preferentially reading from the prefix where +possible. The advantage (compared with just keeping the full pack +file) is that the prefix is a dense store of reachable objects, +improving OS disk caching and the snapshot export performance for +recent commits. In addition, the OS can preferentially cache the +prefix&mapping, which enhances general archive node performance +compared with trying to cache the huge pack file. As baking +operations only need to access these cached objects, their performance +will be more reliable and thus will reduce endorsement misses +drastically. However, some uses of the archive node, such as +responding to RPC requests concerning arbitrary blocks, would still +access the lower layer, so they will not benefit from this +optimisation. The work on improving performance for archive nodes is +planned for Octez v17.

Conclusion

With the context pruning feature integrated, Tezos rolling and full nodes +accurately maintain all and only used storage data in a performant, +compact, and efficient manner. Bakers will benefit from these changes in +Octez v15, while the feature will be included in archive nodes in +Octez v17.

The MirageOS Retreat: A Journey of Food, Cats, and Unikernels

tarides — 2022-10-28T00:00:00-00:00

MirageOS is an OCaml ecosystem to construct unikernels, i.e., minimal operating systems. Here, we write about our social and technical experience at the MirageOS retreat in Morocco, as well as the vibe and wonderful organisational details. To sum up the technical part, we worked on different facets of the MirageOS world: different kinds of unikernels, some groundwork for Raspberry Pi 4 bare-metal unikernels, and a workflow to leverage an existing deployment/orchestrating infrastructure. The MirageOS retreat was amazing!

About Our Journey

Our journey started in Agadir, a Moroccan city right on the coast of the Atlantic sea, just south of the Atlas mountains. In Agadir, we had the best fish in the world (according to some) and amazing "cornes de gazelle," a delicious sample of Moroccan culture.

From Agadir, we went to Mirleft, a small town further south, full of square roads and beautiful reefs. That's where the MirageOS retreat took place. The venue had a kitchen and an amazing cook, a place for computers and presentations, a garden with a small pool, and a rooftop with dusty but nice views of the coast.

+ + + + +

About the Retreat

Both the venue and Mirleft as a whole were extremely inspiring in many ways. One of which included hacking on MirageOS, which was the main reason we came--of course, but we also enjoyed amazing food, saw old and new friends, and had a great time collaborating and creating with MirageOS.

At least once a year since the first MirageOS retreat in 2016 (with a Covid break in 2021), people get together and work on anything related to MirageOS. These retreats provide a great atmosphere, working environment, and everything else that's needed to be productive and to have a wonderful time.

Besides, the retreat is always a nice opportunity to eat our own dog food.

The organiser, Hannes (among others), always makes sure that most of the infrastructure we rely on is running on MirageOS as much as possible. A welcome addition this year was a local opam cache, which allowed us to download and install packages without crushing the data allowance on the SIM card installed on our main access point.

+ + + + +

About MirageOS

MirageOS is an ecosystem that constructs unikernels. In a superficial nutshell, a unikernel is a machine image that contains one process and a minimal set of operating system features the process requires. Unikernels are designed to be secure, efficient, and small. MirageOS unikernels are written in OCaml, a functional, semantically rich and type-safe programming language.

MirageOS can be used in a wide range of settings, like robust reimplementations of core system services and protocols like (DNS, SSH, TLS, and many more), as well as higher level applications like web services. It's also on its way to become a good candidate for bare-metal applications on various chipsets (e.g., a good choice for the Raspberry Pi 4. See also the section below on Implementing a Jack Port Driver)

Our Projects & What We Learned

We worked on lots of interesting things, but let's start with the ones that directly relate to MirageOS.

Deploying Albatross on Nixos: No More iptables Debugging

Albatross is an orchestrator for MirageOS unikernels. It runs on a Linux system and manages unikernels using Solo5. It's made of several services, one of which is the remote TLS endpoint, which accepts requests from the network to manage the orchestrator.

Some of us wanted to run Albatross on our favourite Linux distribution, NixOS, and we hoped to be able to hack around this quickly; however, it turned out to be harder than expected. We learned so much about systemd and networking while doing this project.

A Nix flake (a new way of defining packages, which comes with many rough edges) and a NixOS module are added to the main repository in this PR. +To test that it works and to play with it, we've written a small tutorial that explains how to build a Qemu VM with Albatross and how to deploy a unikernel using the remote TLS endpoint.

Coffee Chat Bot: a Friendly Unikernel for a Friendly Work Environment

Some of us worked on deploying a coffee chat bot as a MirageOS unikernel. Contrary to how it sounds, it isn't a robot that serves coffee (which would be extremely awesome)! Instead, it's a Slack bot that lets people on our company's Slack channel to opt-in for a coffee chat with a colleague. The coffee chat bot then matches each opt-in randomly with another opt-in.

+ + + + +

This bot was already written in OCaml, and it's merely a single process. Due to its nature, it doesn't need to do any super complicated operating system stuff, so a natural question came to us: why not make a unikernel out of it?

So we did.

Making a unikernel out of a relatively simple application sounds rather straightforward. The first step was to get rid of all Unix operations. It's incredible how many small Unix calls we were doing without even noticing. For example, we were using Unix.time all over the place, such as scheduling, providing a seed for the random library, and giving timestamps to our database entries.

The database posed another problem. We had been using irmin-unix, which writes to disk using Unix. To fix that, now we use irmin-mem, which writes to memory. We persist (and inspect) the data by syncing our in-memory database with a GitHub repository. If you're not familiar with Irmin (a MirageOS library), its design follows the principles of the Git design and provides a library called irmin-git to bridge the two.

Providing the network stack needed for the Git (and also for the Slack API) communication is one of the typical tasks the operating system needs do. In our case, that's MirageOS. It has a concept called "devices," which are the operating system features your unikernel might need. Examples of "devices" are network interfaces, network stacks, filesystems (which we didn't need), and monotonic time sources. MirageOS will provide a concrete implementation of such a device at your unikernel's compile time, as long as you declare the device in the MirageOS configuration file config.ml.

The things described were just a small part of our nice, educational journey making a coffee chat unikernel. One more detail that's worth mentioning: the bot now uses httpaf for the Slack API interactions. Before, it was using cohttp, which is already independent from Unix (unlike, for example, the OCaml curl wrapper curly). Porting it to httpaf wasn't technically necessary, but it was a great way to get to know and test the latest "cutting-edge" unikernel features.

Implementing a Jack Port Driver, or How to Make a Unikernel Sing Bare-Metal

We also went bare-metal during the retreat. "Bare-metal" sounds cool, doesn't it? Let us explain what we really mean by it. Often, the way to run a MirageOS unikernel is as follows:

You have a Linux kernel on your machine and virtualize it via a hypervisor such as KVM.
That hypervisor is then abstracted further by a tool called Solo5 which integrates well with MirageOS unikernels.

With this workflow, the communication between the unikernel and the hardware goes over several layers of abstraction. A "bare-metal" unikernel, on the contrary, communicates with the hardware directly, without any interfacing kernel such as Linux. The device we chose to do bare-metal work on is the Rasperry Pi 4 (RPi4).

+ + + + +

So we needed an RPi4 bare-metal OCaml runtime. Luckily, Dinosaure wrote one last year: Gilbraltar. It also dumps the text of OCaml print statements into the UART, which is a technical way of saying that we can send such text over USB (concretely over a USB to serial TTL cable) and see it. Quite useful for debugging!

As you can see, doing bare-metal work is quite restrictive and everything that tends to be taken for granted needs to be implemented, like drivers, for example.

So that's what we decided to do.

Last year, some colleagues already implemented a driver for LED strips and powered our office's Christmas tree with a bare-metal OCaml RPi4! What is cooler than making our bare-metal RPi4 Christmas tree sing? Well, a lot of things are. Anyways, we love music, so we decided to implement a jack port driver.

Jack port drivers on a digital device are an interesting concept. Digital devices are digital, but jack ports expect analog data. One way the RPi4 can handle that is via a concept called PWM: Pulse Width Modulation. The PWM modulates analog signals (i.e., values between 0 and 1) by sending digital signals (i.e., either 0 or 1) really fast.

That modulation is done on the hardware side of the RPi4, concretely on a RPi4 peripheral also called PWM. Peripherals are RPi4 hardware devices that are mapped to specific address ranges in the RPi4's memory. You communicate with them by writing to or reading from those locations in memory. The address range of each peripheral is structured into registers. One example of a register of the PWM is the PWM FIFO, i.e., the hardware queue that stores the data flowing from the program to the jack port.

Our jack port driver does two things--both by writing to and reading from the right places in the PWM memory range.

It can initiate the RPi4 for jack port communication (e.g., it sets the RPi4's clock to the correct frequency at which the port reads data from the FIFO, and it configures the correct modes to ensure the right data flow).
It can send music to the jack port (by writing data to the FIFO--without overflowing it).

To use the new driver, we convert music into the right binary format by simply using ffmpeg. Then we write a program with that music in-memory using the MirageOS tool ocaml-crunch. That program just calls the driver to do the rest and is compiled for the RPi4 target with gilbraltar.

This work is strongly related to MirageOS in three ways. First, the program playing music bare-metal on the RPi4 is a unikernel written in OCaml. Second, the program is compiled with gilbraltar, which forms part of the MirageOS ecosystem and whose design and implementation is based on core tools in the MirageOS ecosystem, such as Solo5 and ocaml-solo5. Third, by adding one layer of abstraction to the jack port driver, we can make it a MirageOS "device," so one could use the driver while also leveraging other MirageOS features that work bare-metal on a RPi4.

Monitoring mirage.io and Chasing Memory Leaks

One of the MirageOS goals is to be able self host our infrastructure. At the retreat, many tools we used were based on the MirageOS ecosytem: a DNS resolver (mirage/ocaml-dns), an opam repository cache (robur/opam-mirror), and a portable file transfer application (dinosaure/bob). It's not a surprise that the official website, mirage.io, is a unikernel itself. However, in the past six months, we experienced two website crashes due to Out_of_memory exceptions. The unikernel is configured to run with 1GB of RAM, so that's a slow running memory leak that requires investigation.

+ + + + +

The question is how to investigate such a leak.

Locally?

The initial attempt consisted in tracing memory allocations using statmemprof while bombarding the server with requests by using benchmarking tools such as ApacheBench (ab) or wrk. statmemprof is an implementation of Statistical Memory Profiling in the OCaml runtime. It enables sampling allocations at a fixed rate and tracing values until they are garbage collected. Using memtrace-viewer, one can analyse the memory usage and see which values are still live when the program goes out of memory, for example. For a unikernel with network access, it's possible to add an endpoint to enable tracing on demand: roburio/memtrace-mirage.

Unfortunately, this setup didn't help us identify the leak. Indeed, we can still expect the server to work fine under normal conditions. Somehow we need to understand which rare event, leaking a small bit of memory at a time, is happening enough times to consume all available memory.

Monitoring the Live Unikernel

Only the Real™ Internet would tell us the answer, so we monitored the live unikernel application. roburio/mirage-monitoring was of great help, as it enables two things:

Reporting application-wide metrics to an InfluxDB endpoint, which can be displayed using Grafana
Changing logs level/metrics sources at runtime

Adding mirage-monitoring to a unikernel was surprisingly easy. It was only a matter of updating the configuration file with some functoria voodoo: https://github.com/mirage/mirage-www/pull/767. At some point, it will upstreamed in the mirage tool so that adding monitoring is a single-line job. The hard part was providing the unikernel two network stacks to expose one to the internet while keeping the other for internal use only.

Next, we set up a typical Grafana deployment using InfluxDB/Telegraf for the metrics input and data storage. Logs were displayed using albatross-client-local.

Chasing the Leak

Now we can see the numbers for the live website. Memory usage, indeed, but also other metrics were included by default, such as the number of established connections in the TCP stack. There we found the source of the leak. Throughout the day, the number of established TCP connections kept increasing.

Finally at runtime, we temporarily changed the TCP stack's log level to debug, monitor the logs, and wait for the moment where the number of established TCP connections would increase without decreasing afterwards. These logs described what was going on in the TCP stack at the exact moment the connection leak happened. At this point, we figured out that it occurred when a client connected to the server but fail to perform the TLS handshake, so the server dropped the connection without closing it--hence leaking it forever.

Here we go: one less leak.

Next Steps

Matching logs and metrics to inspect them together has proven to be very useful. We used Grafana for metrics, so the next step would be to also provide logs because Grafana supports structured logging through the Loki logs aggregation system.

MirageHole - A Unikernel DNS Resolver with Holes

One way to stop web trackers, advertisements, and malware is to block access to sites known to contain such things. A popular approach is through browser extensions like AdBlock and Privacy Badger. Another approach known as a DNS sinkhole involves installing a local DNS server that resolves bad domains to an invalid IP address. This approach has the advantage of working across different operating systems, browsers, and devices (laptops, smartphones, smart-TVs, etc.). For an added bonus, it can also save network bandwidth.

Another project initiated during this year's retreat was to implement Mirage-hole: a DNS sinkhole running as a Mirage Unikernel. It was inspired by Pi-hole for the Raspberry Pi. Starting from a DNS-stub example from dnsvizor (and after a bit of network debugging), we got a unikernel running that would block a single selected domain. We then extended this to fetch and parse a blocklist at start-up. Next, we worked on integrating a little webserver to serve statistics about the requested and blocked domains. Overall, the project was a nice opportunity to talk to and learn from several MirageOS contributors, and it served as a nice tour-de-force of several MirageOS networking libraries.

Tarides Map - Serving the Tarides Geographical Distribution in a Unikernel

Tarides Map is a project intended to show the geographic distribution of all Tarides collaborators as a website. At the retreat, we explored deploying the site in a unikernel. To do this, we had to decide how to serve the files on the server and integrate it into a unikernel. We had two options use ocaml-crunch or Docteur.

We initially used Docteur due to an inspiration from a different project called Pasteur, which uses Docteur and is deployed in a unikernel as a static site, which was exactly what we were aiming to do with Tarides Map. However, integrating Docteur into the project proved to be more difficult than we had expected. One reason was that Solo5 isn't currently supported on MacOS, the operating system used to write the project at the retreat. After compiling to Unix instead and numerous hours debugging, we were eventually able to generate the disk image; however, we still had issues deploying it in a unikernel, so we decided to try using ocaml-crunch instead.

Using ocaml-crunch proved to be a more straightforward option. We merely had to move some files around so that the directory structure could be turned into a standalone OCaml module to serve the file contents without requiring an external filesystem to be present. After doing this, we were successfully able to deploy the site here.

What We Dreamed About

Another very interesting part of the retreat were the dreaming sessions organized by Hannes. The central idea behind this exercice was to allow ourselves to dream about how we envision the MirageOS project in the future, no matter how untangible and seemingly unrealistic. We talked about those dreams in two sessions.

The initial session revolved around gathering these dreams and ideas, without discussing how to achieve them, and let our mind go free with what we wanted to accomplish with MirageOS. Often times, those dreams would be shared with other participants. Some dreamed about replacing their whole software infrastructure by MirageOS, if not their main operating system! Others dreamed of artistic applications for Mirage, like using it as a backbone for musical endeavors.

The subsequent session revolved around how we could reach those dreams. This facilitated a more practical discussion around the challenges we may face along the way. Interestingly enough, in some instances, it turned out some dreams were either already achieved (like reverse-debugging Solo5!) or were close to being achievable.

A beautiful example of the attendees' dedication is that it did not take long for some to start working on projects like MirageOS-OS, a hypervisor for MirageOS unikernels and written with MirageOS, or to successfully implement a jack port driver for the Raspberry Pi 4, bringing us closer to MirageOS powered synthesisers and to MirageOS midi interfaces!

+ + + + +

Inventing OCamlwave, Serenading Cats, and Christening Dogs

As mentioned above, the retreat was extremely inspiring, even with respect to topics less related to MirageOS than the ones mentioned here. The one we're most proud of is our Mirleft MirageOS EP that contains five tracks (five in the spirit of Solo5 and OCaml 5.0, of course). Its genre might be better described as OCamlwave! On our EP, you will find many musical oddities ranging from an on-premise recorded drum solo (with glasses, cloth-racks, and flip-flops) to a cat-powered cover of Mr Sandman (as an hommage to our time singing to Morroco's many, many cute cats.) to the occasional dramatic rendition of controversial pull requests on the OCaml compiler.

+ + + + +

However, not everything in Mirleft was about music and animals. Some things were also about the beauitful waves.

Mirleft is a paradise for surfing, both for beginners and advanced surfers! We went to a nice sandy beach with perfect conditions to get started with surfing. Advanced surfers would probably go to one of the reefs for surfing, which we, in turn, found amazing for a peaceful walk, sometimes with and sometimes without company from a street dog we christened null.

+ + + + +

And, well, talking about null (apart from naming street animals), we also had plenty of other computer science related conversations at the retreat. All of them were extremely enriching! A couple of examples include exception backtracing in LWT programs and BGP intrinsics.

Thanks for All the Fish!

As this lengthy report can attest, our experience was an amazing one for all. The MirageOS Hack Retreats are always an otherworldly space, where amazing individuals gather to exchange thoughts and create new (and better) software. Friends are made along the way, some bugs are fixed, new ones are found, and great new ideas emerges.

This very special sense of community is rare, so we would like to thank everyone who organized, attended, and tended to the event. Thank you to our delightful hosts, who've been with us since the first retreat in 2016! Thank you as well to Hannes and Robur for organizing those retreats and spending time instilling the same inspiration in the great project that is MirageOS! Finally, thank you to old and new friends, as well as old and new MirageOS hackers, for this amazing week of happy banter and hacking!

PD: Some of the pics in this post were shared among us via bob, a MirageOS unikernel to share files.

Up-to-Date Online Documentation

tarides — 2022-10-20T00:00:00-00:00

Into the Fire

The OCaml ecosystem relies on various resources and infrastructure such as ocaml.org, OCaml Docker images, opam-repo-ci, that are built and deployed using OCurrent. OCurrent is a library to express workflows and keep things up to date. As many of these projects are created using the same technology, it was interesting to centralise the documentation as it was spread throughout the various repositories. This post is about how we used OCurrent itself to automate this problem. We think it might also demonstrate how you can use OCurrent to automate some of yours!

Can't Keep My Eyes Off You

Before digging into the logic, it's essential to thoroughly define the problems in the documentation. The first problem was that the documentation lives in many GitHub repositories. Indeed, to make sure we update it whenever we modify the associated code, we keep the documentation closest to the code. The result is a repository organisation like this:

+ + + + +

It's not a good judgement to count on humans' actions to monitor changes in all these repositories. As they fluctuate on their own time, we can't expect maintainers to backport documentation changes to the ocurrent.org website for each modification. To say it more technically, we need to track many files and keep them up to date. These actions should also update incrementally which matches with OCurrent nicely.

In addition, this documentation needs to stay up to date. Even if we centralise the documentation automatically, we must rebuild it regularly and fetch the changes from the repositories we track. Otherwise, the documentation will start to be outdated quickly. This is the opposite of what we want.

Furthermore, the system has to scale and be updated easily. Indeed, we would like to have the possibility to introduce new documents and repositories without having to install more applications. For instance, it would be beneficial to simply make a pull request somewhere.

In the next section, we will focus on the OCurrent pipeline design, which will automate our tasks and solve these problems.

Here I Dreamt I Was an Architect

The project is composed of several blocks we want to write to achieve our work:

Fetch files from GitHub
Rebuild a subset of the code
Make the system modular
Store and deploy the data easily

One aspect that will make our work a bit easier is to have it all concentrated in the same place, GitHub. As OCurrent provides a plugin to fetch information from GitHub, current_github, we don't have to worry about it. Furthermore, everything is cached thanks to OCurrent itself. We don't have to care about the incremental build. The only requirement is wisely choosing the data we want to cache.

Our architecture uses a trackers.yml file describing how the pipeline should interact with our heterogeneous repositories. It describes the files we want to track and where we would like them in the final website structure. The configuration gives a way to achieve modularity at a low cost, as we only have to open a PR on the repository that contains the tracker file to update them. Additionally, it allows us to track the repositories we want quickly. Once it's followed, we don't have to worry about the monitoring, as OCurrent can be set to rebuild stuff at the regular cycle. In our case, we want to control every week that the code hasn't mutated. In the present version of trackers.yml, we can specify the files we want to copy and the indexes we want to create to build our structure. This file is stored in the repository on which the GitHub App is installed.

Another critical component in the architecture is handling new files from the remote repository and integrating them into the website structure. This element is in charge of moving the piece from one part of the system to another. Moreover, it will have to ensure the paths are consistent and fail if not.

The last item must push the code to a specific Git repository because we decided to use GitHub Pages to store the website. To avoid issues with account management, it needs ssh to get access to a specific repository.

In the end, the pipeline design would look like this: + + + + + +

Now that we have our workflow let's see how it is implemented in practice!

This is How We Do It

In this section, we will focus on the way to implement this infrastructure. We won't view all the elements in detail, but we will try to concentrate on the most important ones, like how to create a custom ocurrent component and chain them together to build a pipeline.

`current_github`

Let's focus on a standard structure in an OCurrent project: the way to get the HEAD of a branch on GitHub and fetch the commit with Git. In the related code, we find the HEAD, then ask GitHub to give us information about the HEAD commit on the default branch and finally get the content with Git (it returns the related commit):

let fetch_commit ~github ~repo () =
+  let head = Current_github.API.head_commit GitHub repo in
+  let commit_id =
+    Current.map Current_github.Api.Commit.id head
+  in
+  let commit =
+    Current_git.fetch commit_id
+  in
+  commit
+
+let main =
+  let github = (* GitHub App code *) in
+  let commit = fetch_commit ~github ~repo () in
+  (* Use the commit code *)

The documentation of current_github and current_git is available online.

Fetching the Files

As we know how to extract data from GitHub, applying the process to various repositories will be easy. It can be noticed that the commit element is of type Commit.t Current.t. To work with Current.t, we need to "unwrap" the object with specific functions like map and bind. This post does not present how to load the content from a Yaml file. We assume that we get a selection list Current.t, where selection is defined as:

type selection = {
+  repo : string;
+  commit : Current_git.Commit.t Current.t;
+  files : 'a list;
+}

It contains the source repository, the commit associated with the specified branch, and the list of files to monitor from this repository.

To git clone the content, we must apply the fetch_commit function.

Copy the Content

In this subsection, we will see how we can define a custom component and how to make it interact with the rest of our code.

The component is in charge of fetching the content of the files from the source directory and storing it in memory. To trigger the action only when the content changes, we will define a Current_cache element. Thanks to OCurrent, the content is cached and only rebuilt on change or request.

It manipulates some File.info (source, destination, …) and produces a File.t when the content is read. File.t is simply a:

type File.t = {
+  metadata: File.info;
+  content: string list;
+}

Our file is represented as a string list, as we need to be able to add more information. We know the size of the files is limited, so it is not an issue for us. +The component is defined as a Current_cache.BUILDER with whom the signature looks like this:

module type BUILDER = sig
+
+type context
+
+module Key : sig
+  type t
+  val digest:
+end
+
+module Value : sig
+  type t
+  val marshall : t -> string
+  val unmarshall : string -> t
+end
+
+val build :
+  context ->
+  Current.Job.t ->
+  Key.t ->
+  Value.t Current.or_error Lwt.t
+end

As the Value and the Key modules only use functions to manipulate JSON, we can focus on the build function definition:

let build files job { Key.commit; Key.repo; _ } =
+  Current.Job.start job ~level:Current.Level.Average >>= fun () ->
+  Current_git.with_checkout ~job commit @@ fun dir ->
+  extract ~job ~dir repo files
+  >>= Lwt_result.return

It creates a temporary directory with the content fetched from Git. Then, it extracts the data as a File.t and returns the result. The interesting detail here is Current_git.with_checkout fn. It is used to copy our code somewhere in the system temporarily. Current.Job.start is just some boilerplate code to start a job asynchronously.

Consequently, we can give the builder a functor to construct our cache system. Moreover, we create a function associated with it thanks to the Content module newly created:

module Content = Current_cache.Make (Content)
+
+let weekly = Current_cache.Schedule.v ~valid_for:(Duration.of_day 7) ()
+
+let fetch ~repo ~ commit files =
+  Current.component "fetch-doc" |>
+  let> commit = commit in
+  Content.get ~schedule:weekly files
+    {content.Key.repo; Content.Key.commit }

We specify the date when the cache is invalidated to trigger the rebuild at least every week.

Build & Deploy

In this last subsection, we discuss how to write all the files stored in the cache to the right place in the filesystem. We use hugo to build the website and git with ssh to deploy it. As we expect the information to be cached, we build a Current_cache module again, where the build function is:

let build { files; indexes; conf } job { Key.commit; _ } =
+  Current.Job.start job ~level:Current.Level.Average >>= fun () ->
+  Current_git.with_checkout ~job commit @@ fun dir ->
+  write_all job dir files indexes >>= fun () ->
+  Lwt_result.bind (hugo ~cwd:dir job) (fun () ->
+      let f cwd =
+        let commit = Current_git.Commit.hash commit in
+        deploy_over_git ~cwd ~job ~conf dir commit
+      in
+      Current.Process.with_tmpdir f)

In this context, the pipeline creates an indexes file as _index.md. It's used by Hugo to build the directory structure. This function uses the same Current_git.checkout process to create a temporary directory containing the website's skeleton. All the work is done in the deploy_over_git function, but this is not relevant to go further in detail. The component writes all the File.t.content to the destination specified in their metadata. Once we have successfully written them, we generate the website with hugo --minify --output-dir=public/. Last but not least, we copy the content of the public repository to a fresh temporary one, so we can add the files with a git init and push our work to GitHub. Finally, on the target repository, GitHub Pages will deploy the website.

And voila, our website is up-to-date and online!

Happy Together

This blog post has described how we handle our distributed documentation and centralise it on our website. We have seen how to use some Current_* plugins and how to write our own. It was also the occasion to speak about various OCurrent structures.

If you are curious, you can check the code in the ocurrent/ocurrent.org repository. Feel free to look at the ocurrent.org built with this pipeline. The description of the pipeline is also available in the bin repository.

Porting Charrua-Unix and Rawlink to Eio

tarides — 2022-10-19T00:00:00-00:00

This article describes the porting of the DHCP daemon charrua-unix and its companion library rawlink to Eio for the upcoming OCaml 5 release. Before we get started, it makes sense to briefly describe what DHCP is and how we use it in production.

What is DHCP?

DHCP stands for Dynamic Host Configuration Protocol, and it's described in RFC2131, RFC2132, and others. It was first published in 1993, so it's considerably old, yet very much alive in virtually every network these days—from your home, to your office, to your ISP Wide Area Network.

When your computer, laptop, phone, or any IP-connected device boots up or changes network, it requests network parameters via broadcast. These parameters are requested and answered via the DHCP protocol. The common/minimum parameters a client requests are:

An IPv4 address
An IPv4 gateway
The address of a DNS resolver

This is enough to get connectivity in most networks. DHCP can also provide many extra parameters, but they are outside of the scope of this document.

What is `charrua-dhcp`?

charrua-dhcp is a DHCP library suite written in pure OCaml. You might not know it, but if you have ever used Docker Desktop, be it on Windows or macOS, you're a user of charrua-dhcp already !

In Docker Desktop, a complete Linux VM is run in the background in order to be able to run Docker containers. This VM needs to acquire network parameters from the host operating system, and this is done via charrua-dhcp. You can check more details on how OCaml and charrua are used to power Docker Desktop in this article.

charrua-dhcp is also the standard DHCP implementation used in Mirage OS, both when used as a server or a client, and perhaps more importantly, it is used on high profile, critical cases, like the home network of yours truly. It is a stable and tested library that has been in use for years, and it has also been put to the challenge against Crowbar. See more details in this article by Mindy Preston.

charrua-dhcp is split into charrua-core and charrua-unix:

charrua-core implements the DHCP server and client logic in pure OCaml, as well as providing serialisers and deserialisers for the protocol wire format. It also provides a textual configuration interface, like ISC-DHCP does.

When we say pure OCaml, we mean it! charrua-core is purely functional and doesn't produce anything via side-effects; therefore, it also does not perform any kind of I/O.

charrua-unix implements the effect-full bits, and it does I/O, feeding incoming packets to charrua-core and sending out replies given by charrua-core.

The idea is that charrua-core has the complex DHCP logic, while charrua-unix does the basic things: logging, sending/receiving packets, making sure the environment is secure, and so on.

The name charrua is a reference to the seminomadic tribe Charrúa from what is today Uruguay, Argentina, and southern Brazil. The rationale is that DHCP serves parameters to roaming (nomadic) clients.

What is `rawlink`?

DHCP is not an IP protocol. It sits above the Ethernet layer, which means a DHCP application must be able to craft and receive the full Ethernet packet, not just the layers above IP.

Each operating system provides a slightly different mechanism on how to accomplish this. Linux provides a special socket family called AF_SOCKET, whereas BSDs (OpenBSD, FreeBSD, macOS...) and most other Unix systems provide the same via BPF.

rawlink is an OCaml library with C stubs that abstracts these differences away. You get a link on a network interface, which you use to craft and receive full Ethernet packets, bypassing most of the operating system network stack. In other words, rawlink allows you to work with raw packets on an Ethernet link.

+ + + + +

What Changes in OCaml 5?

OCaml 5 provides two main new features:

Parallelism
Effect handlers

Parallelism makes little sense on a slow, control protocol like DHCP, so we don't use it and it's not the focus of this article.

Effect handlers allow OCaml programs to write non-blocking code as if they were blocking.

Until OCaml 5 and effect handlers, the common way to write non-blocking code was through Lwt, a concurrent programming library for OCaml. Lwt provides a concurrent scheduler and a monadic style of writing programs through promises. With it, the program becomes a long string of binding promises.

One issue with Lwt is that it's very "infectious," and as soon as you add the first Lwt promise (called "thread" in Lwt lingo), the whole code must now behave as a promise as well. Another issue is that the monadic programming is somewhat syntax heavy, so it can clutter the code. Since the promises are allocations themselves, it can also negatively affect performance. Lwt is a great library, but with OCaml 5 and effects we can do better.

With OCaml 5 and effect handlers we can have the best of both worlds. We can write non-blocking code in a blocking style without the monadic clutter imposed by Lwt. The library we are proposing to replace Lwt in OCaml 5 is Eio, which takes full advantage of the effect system, as well as providing a framework to express parallelism.

Lwt vs. Eio

This code snippet is the main function of charrua-unix, using Eio (left) and Lwt (right). We can summarize what is happening as follows:

1 - We read a packet from the network. +2 - We feed the packet to charrua-core, which then gives us a possible Reply (reply, db), the packet to be sent out and the new DHCP database state, respectively. +3 - We send the reply out and loop for more packets.

It's a fairly simple code, but it shows how much less cluttered the Eio version can be by removing all Lwt decorators. Another nice advantage is that if we were to write a blocking version of the same code, we would only need to change Eio_rawlink.{read,write}_packet to Rawlink.{read,write}_packet as their signatures remain the same, something impossible with Lwt.

+ + + + +

`rawlink` and Eio Switches

rawlink uses a file descriptor that Eio knows nothing about, so in order for us to use Eio with it, we want to attach an Eio.Flow.t to the file descriptor. An Eio.Flow.t is an Eio abstraction of a bidirectional socket, even though it was designed mostly for a STREAM-like socket in mind, the semantics fit rawlink case. We do this in Rawlink_eio.opensock:

let open_link ?filter ?(promisc=false) ifname ~sw =
+  let fd = Rawlink_lowlevel.opensock ?filter:filter ~promisc ifname in
+  let flow = Eio_unix.FD.as_socket ~sw ~close_unix:true fd in
+  { flow; fd; packets = ref []; buffer = (Cstruct.create 65536) }

Rawlink_lowlevel.opensock is a call into the actual C stub that returns a BPF or AF_PACKET descriptor, we then create the Eio.Flow.t with Eio_unix.FD.as_socket.

Two things appear out of the ordinary in the flow creation call: The sw (Eio.Switch.t) and close_unix arguments, in order to make sense of them we have to understand what an Eio.Switch.t is.

A long standing issue with Lwt was "how to make sure my file descriptors are not leaked if something goes wrong." Eio attempts to solve this by forcing each Eio.Flow.t to belong to a Eio.Switch.t. You can't create a Eio.Flow.t without giving it a Eio.Switch.t, so this is what the Eio_unix.FD.as_socket does. Since Flows are also attached to normal file descriptors, Eio.Switch.t also takes care of them.

An Eio program creates one or more Eio.Switch.t in order to attach a Eio.Flow.t to it. An Eio.Switch.t can also be nested, creating a tree-like structure, as every new Eio.Switch.t becomes a child of its parent Eio.Switch.t. When an Eio.Switch.t terminates, either succesfully or by some exception, all of its children Eio.Flow.t are also terminated, automatically closing the file descriptor and guaranteeing we don't have a descriptor leak.

close_unix tells Eio to call close(2) when the Eio.Switch.t terminates.

Imagine a TCP server where each client has at least one dedicated Eio.Switch.t, and some of these clients create additional Eio.Switch.t to handle a specific unit of work:

+ + + + +

Conclusion

Both Lwt and Eio provide means to achieve concurrency, but they only provide parallelism with Domains. Lwt uses monadic-style promises to achieve concurrency, which pollutes the code and makes it harder to reason about it. Eio makes full use of the new effect handlers and Domains of OCaml 5, providing concurrency and parallelism while maintaining the same programming style of synchronous blocking programs.

Eio is a library that aims to replace Lwt, but with a more modern style and feature set. It provides abstractions for sockets, fibers, streams, flows, and more.

To review,charrua-unix is a feature-packed, yet simple DHCP server implementation for Unix systems based on the OCaml library charrua-core.rawlink makes it possible to read and craft Ethernet packets on most Unix-like systems through an easy-to-use library.

It's relatively easy to port rawlink to Eio by attaching an Eio abstraction of a bidirectional socket, namely Eio.Flow.t, to the file descriptor.

We hope you enjoyed this article and found it helpful. As always, if there are any questions or concerns, feel free to reach out.

OCaml's Platform Installer Alpha Release

tarides — 2022-10-18T00:00:00-00:00

Yesterday we announced the OCaml 5 beta release, and today we're excited to introduce the OCaml Platform Installer! The OCaml Platform is the recommended toolchain when working with OCaml. This new installer enables programmers to quickly set up the OCaml developer environment, so they don't need to waste precious coding time with a lengthy installation process. If you come across any obstacles, the Platform team encourages you to open a GitHub Issue.

We've also updated the state of the Platform, making several important changes like promoting odoc and OCamlformat from Incubate to Active. We have notified the OCaml Community about the Platform Installer's alpha release, so you can read about all the new changes and the simple installation process on the OCaml Discuss post.

Stay tuned to our blog as well as our Twitter feed to get the latest updates on the OCaml 5 release!

OCaml 5 Beta Release

tarides — 2022-10-17T00:00:00-00:00

Back in June, we announced the OCaml 5 alpha release, and today we're excited to announce the first beta release! Now is an excellent time to test it and report positive or negative feedback on your projects (i.e., did it work, did you see impressive performance speed up, did you have issues finding documentation, etc.)

This beta version stabilised several opam packages, fixed several small internal runtime processes (especially the systhreads library), and tweaked the Domain and Effect interface, just to name a few improvements. This version also enables you to update your libraries and software. See the post on the OCaml Discuss forum for installation instructions and more information. While you're there, join the growing and vibrant OCaml community!

The full OCaml release is expected by the end of the year. Just in time for Christmas! Perhaps more importantly, in time for the new Advent of Code calendar, so you can play around with OCaml 5 with Multicore support.

Real World OCaml Book Giveaway!

tarides — 2022-10-14T00:00:00-00:00

Real World OCaml is a fantastic book on OCaml and functional programming – a great resource for beginners and experienced users alike. At Tarides, we want to support new learners of OCaml as much as we can, making it easier for people to become part of the vibrant community surrounding the language. +Tarides is proud to announce that we are sponsoring the Gold Open Access release of Real World OCaml, 2nd Edition by Yaron Minsky and Anil Madhavapeddy! It’s published by Cambridge University Press, and Tarides is making it possible for everyone to download the book to their local device. You can also receive free copies of the book (see below).

Accessible to Everyone and Free Copies

Since its first release in 2013, the book has been a fantastic resource for members across the community. On the Open Access release, its authors said, “As long-standing members of the open-source community, we are excited to see our work made more accessible for all users of OCaml.”

The authors and publisher have generously agreed to give away some physical copies of Real World OCaml. They really want to reward the amazing and active members of the community who are so engaged with the book. That’s why ten people who get a PR merged with a suggested improvement to the book on GitHub will receive a free copy of Real World OCaml! Just email rwo@tarides.com when your PR has been merged, and we'll let you know if you're one of the lucky 10 who receive a book!

Gold Open Access

Until recently, you could read Real World OCaml online, but for offline access, you had to rely on the printed version. With Gold Open Access, you can now download a PDF to your local device for easy access at any time.

Whilst this expanded access will benefit everyone, one group that we are particularly excited to support are new learners. OCaml 5.0 is right around the corner, and we want to make the entry to OCaml for new users as easy as possible. Making the book more accessible also aligns with Tarides’s inclusivity goals; lowering the barriers to entry into the community will encourage greater participation among people who otherwise would not have had the means to join.

On this topic, David Tranah, the editorial director of Mathematical Sciences and Information Technology at Cambridge University Press, shares his unique insight into the benefits of Open Access: “Gold Open Access publishing allows anyone, anywhere, who can connect to the internet to stay up-to-date on the latest research. This in turn drives innovation and leads to new discoveries.”

We of course encourage anyone with the means to purchase a physical copy of the book, as it’s the result of a lot of hard work and dedication, and it comes in a beautifully printed physical edition.

Real World OCaml

The first version of Real World OCaml was written by Yaron Minsky, Anil Madhavapeddy, and Jason Hickey. Since its release, several contributors have improved on the original text, adding new examples, correcting errors, and expanding on chapters. The second edition was published in 2021 by Anil and Yaron and includes the most recent improvements and changes for an updated version of the book.

The book itself covers several aspects of OCaml, from fundamental concepts like functors and objects, to different tools and techniques, including the OCaml Platform and JSON, as well as a section on the compiler and runtime system. It takes its reader on a journey through OCaml moving from basics to increasingly advanced topics, making it the perfect companion for anyone regardless of their level of OCaml.

Over the years, Tarides has supported Real World OCaml by contributing to the book’s tooling infrastructure. Some of that work has transformed into standalone community projects like MDX that help to improve all OCaml documentation.

The book also has a mutually beneficial relationship with OCaml.org, as Thibaut Mattio (currently leading the community redesign effort of OCaml.org) explains: “There are crosslinks between Real World OCaml and V3 of OCaml.org. The new package documentation site is a great accompaniment to Real World OCaml, and there are multiple links to it embedded within the book for API documentation.” This is just one example showing how Real World OCaml is used in projects and interacting with new content in a productive and useful way.

About the Authors

Anil Madhavapeddy is Professor of Planetary Computing at the University of Cambridge and a fellow of Pembroke College. He has a wide range of experience, having worked in industry (NetAPP, Citrix, Intel), academia (Cambridge, Imperial, UCLA), and open source (OCaml, OpenBSD, Xen, Docker). +Joining Jane Street in 2003, Yaron Minsky is to thank for introducing the company to OCaml. Founding the firm’s quantitative research group, he managed the transition of all its core infrastructure to OCaml, ultimately making it the world’s largest industrial user of OCaml. Minsky has also been an avid lecturer, blogger, and writer on the topic of programming, publishing articles in Communications of the ACM and the Journal of Functional Programming.

Summing up their thoughts on the Open Access upgrade, Anil and Yaron say: “Open Access has been shown to encourage the usage of a particular work, resulting in increased citations and public engagement. We are excited for what this move will mean when it comes to greater accessibility for users of OCaml worldwide: making it easy to use excerpts from our book in new projects, encouraging new learners, and supporting teachers in their work.”

8 OCaml Libraries to Make Your Life Easier

tarides — 2022-10-12T00:00:00-00:00

OCaml is a statically-typed programming language that emphasizes readability, programmer efficiency, and semantic clarity. This powerful and efficient language has been gaining popularity among developers. The growing adoption of OCaml is because it's fast, type safe, and secure. It can be used in industries where performance and security matter, like IoT, Data Analytics, or financial services.

Like any other programming language, there are numerous libraries available for OCaml that can make your life easier as a developer. In this article, we will explore some of the top OCaml libraries that will help streamline your workflow and boost your productivity as a programmer.

All these libraries and tools are open source, so they’re distributed under a free software licence.

A Few Helpful OCaml Libraries

`Lwt`

Lwt is a library for writing asynchronous code. It provides many helpful abstractions for writing asynchronous code such as promises and futures. This is a very useful library for writing network applications, like web servers. That said, the Eio library in the forthcoming release of OCaml 5.0 might become preferable!

Dream

Dream is described as a "tidy, feature-complete Web framework" on OCaml.org. Dream has a simple programming model where web apps are merely functions, and it supports TLS, WebSockets, and GraphQL. Plus it has cryptography helpers! It's easy-to-use and documented all in one place. The entire Dream API is available there, where you can also find many examples.

`Cmdliner`

This library is used by several packages to build command line tools, which is beneficial when you want to write an executable in OCaml. Cmdliner gives programmers a simple, compositional method for turning command line arguments into OCaml values. Not only can you then pass those values to functions, Cmdliner can automatically handle syntax errors, help messages, and UNIX man page generation as well.

Alcotest

This colorful framework performs simple unit tests on a simple interface. Alcotest only displays faulty runs at the end of the output, along with full logs for your inspection. The straightfoward, expressive query language makes it easy to select which tests to run, and the results are displayed in a fun rainbow of colors.

`base`

Although the standard library is somehow minimalist, multiple extensions exist to make programmers' lives easier. For instance base, created and maintained by Jane Street, is used to develop critical applications by industrial users. baseis written in pure OCaml and has no dependencies other than the OCaml standard library. The base library is useful for building many applications. Read more about base in the book *Real World OCaml.

Yojson

Yojson is an OCaml library for creating and reading JSON data in OCaml. JSON is a data format that is commonly used in web applications. The Yojson bindings can be used to easily generate and parse JSON data in OCaml.

Notty

This interesting OCaml library enables the user to write declarative terminal UI. Notty is based on a notion +of composable images, and it delivers a more simple and expressive model than the basic terminal programming. Engineers know that programming terminals are tedious, so Notty makes it enjoyable!

`ppxlib`

PreProcessor eXtensions, or PPX for short, are used for meta-programming, like for generating boilerplate or for extending the OCaml syntax. PPX act on the AST and are integrated into the language via two AST features: attributes and extension nodes. ppxlib is a set of tools and libraries that enables programmers both to write and use PPX. See this Tarides blog post on how to write PPX and this official guide on how to use PPX.

Conclusion

The tools available in OCaml make it easy to prototype new applications and build production-quality software. In fact, the full release of OCaml 5.0 with Multicore support is on the horizon, and the alpha version has already been released.

OCaml libraries help you write beautiful, elegant code in this powerful and versatle language. There has never been a better time to give OCaml a try, and now you know there are beneficial libraries to help you code. The libraries covered in this article are just a few examples. For more information, please visit ocaml.org

ICFP 2022 Review

tarides — 2022-10-10T00:00:00-00:00

After two years of online conferences, it was fantastic to have ICFP 2022 in person. The conference organisers had done a fantastic job adjusting to online conferences, but nothing beats the hallway track for meeting new people and catching up with old friends. This year, Slovenia's capital hosted the event. Ljubljana was a beautiful city to visit, with plenty of classic European architecture and even a castle to explore.

My conference schedule was packed. I had a talk to present on Friday and five preceding days of conference talks to attend. The first three days were a whirlwind of talks (some on the edge of my understanding) and hallway track conversations.

+ + + + + +Image by Yaron Minsky of Jane Street

OCaml Reaches for the Stars

Although it happened mid-week, I want to start with KC Sivaramakrishnan's keynote, Retrofitting Concurrency – Lessons from the Engine Room, as it was definitely the highlight of ICFP. He covered the full story of introducing parallelism and concurrency into OCaml, along with references to papers published along the way. Watch the video of his keynote. The "Where do we go from here?" slide ties together the effect system with targeting Javascript, modal types to avoid heap allocations, unboxed types to control memory layout, and Flamba2 for aggressive compiler optimisation. This is hugely exciting for the OCaml community. It brought together many threads of work into a coherent picture.

+ +

There was a real buzz afterwards from both OCaml and Haskell people I talked with. It set the stage for the ML Workshop on Thursday, which included Efficient and Scalable Parallel Functional Programming Through Disentanglement, Unboxed Types for OCaml, Module Shapes for Modern Tooling, Stack Allocation for OCaml, and Boxroot, Fast Movable GC Roots for a Better FFI, picking up themes from the keynote. The full playlist for the ML Workshop is available on YouTube.

+ + + + +

Thursday also featured the OCaml 5.0 for the Working Programmer tutorial, presented by my colleague Sudha Parimala (above) and Marek Kubica at Tarides. There are slides and exercises to work through to help you understand effects and parallelism in OCaml.

Friday: A Full Day of OCaml

The final day was dedicated to OCaml Workshops. KC kicked off the first session with a keynote on the "here and now" of OCaml 5.0. His presentation addressed developers' frequently asked questions when moving from sequential OCaml 4 to 5.0, discussed the details of the merge process, and included a deep dive of the developments since the merge of Multicore OCaml early this year. His talk concluded with a call to action, encouraging OCaml developers to start migrating to OCaml 5.0, even if they do not immediately plan to use the new concurrency and parallelism features.

The keynote was followed by Jan Midtgaard, Principal Engineer at Tarides, talking about a number of testing techniques and tools for concurrent programs that Tarides has developed for OCaml 5.0. This was followed by Deepali Ande, KC's student at IIT Madras, presenting a novel way to enable different schedulers written using effect handlers to communicate with each other.

These talks ended up being so popular that there was standing room only, so during the coffee break after the first session, the ICFP organisers announced to everyone that the remaining OCaml presentations would be in "the fanciest theatre ever," an amphitheatre known as the Štih Room (below).

+ + + + +

I presented our collective work on bringing OBuilder to non-Linux platforms. OBuilder is the underlying library responsible for providing sandboxed build environments for ocaml-ci, opam-repo-ci, and opam-healthcheck. The talk covered the architecture of OBuilder, showing how it gets used in our multi-archtecture cluster of build machines. I also reviewed the Linux implementation that uses native Linux containerisation technology, like runC and cgroups. Then, moving onto the implementation on macOS, I demonstrated using user isolation to provide sandboxing with some file system tricks, followed by the implementation on Windows using Docker for Windows, which during testing found a number of interesting bugs in LWT and GNU Tar. The full details are available in the Extended Abstract and on GitHub https://github.com/ocurrent/obuilder/.

The OCaml Workshop also featured an impressive back-to-back presentation from David Allsopp on opam's CLI compatibility work and the upcoming opam 2.2 features. He also covered how to make the OCaml compiler relocatable and how that will allow for fast switch creation in opam. Congratulations to David for making such a polished and well-received double show!

+ + + + +

Functional Programming Development

Although OCaml certainly stole the show after KC's keynote, earlier in the week I attended several fascinating workshops, a few of which I outline below. There was also a strong theme of Effects talks throughout the third day of ICFP, which I plan to come back to and review the papers in more detail. Below are some highlights, but the full Haskell Implementors Workshop (HIW) is available on YouTube.

Haskell

On Sunday, I spent my day in the Haskell Implementors Workshop, which kicked off with the "State of GHC" talk from Simon Peyton Jones. Simon covered all the new and important developments in 9.4 and 9.6. For me, the complete overhaul of GHC’s Windows support, including many fixes in WinIO, refactoring of GHC's error messages, and the ongoing work to upsteam GHCJS and WebAssembly backends into GHC, were the most exciting changes. The other takeaway was how Cabal development has been overhauled with a new team managing the project and the resulting acceleration of improvements making their way into Cabal, starting from Cabal 3.6. That, combined with the significantly improved Haskell LSP support, means Haskell tooling is in a great place.

GHC & Racket

Alexis King's talk, "A Look Across the Pond: A Comparison Between GHC and Racket Compilation Models," was a great highlight on how Racket tooling works and how Cabal could be further improved, and perhaps how we could improve opam. The key idea was that Racket uses a set of core data structures to represent packages and provides functions across those data structures, with the end-user tooling being a very thin wrapper around these functions. Alexis demonstrated how to query and manipulate the set of installed packages within Racket in a way that's impossible with current Cabal. This is an intriguing idea that hopefully gets some attention, and I wonder if the more dynamic representation available in Racket makes this easier compared to Haskell.

GHC & Mu

My second recommendation is "Compiling Mu with GHC: Halfway Down the Rabbit Hole" by Georgo Erdi, which covered the effort to port Mu to reuse the GHC compiler frontend and backend as much as possible. Currently Mu uses a custom compiler for its strict Haskell variant that includes MultiParam Typeclasses and Functional Dependencies, along with a variation on Type Families. I like hearing about compiler engineering efforts that need to handle existing codebases and think through the trade-offs. The choice of Functional Dependencies and Multiparam Typeclasses is a sweet spot in the design space for typeclasses, with Purescript choosing a similar approach.

OCaml Reception

Overall the ICFP had a huge buzz of excitement around OCaml 5.0 featuring Multicore support and effects. The palpable enthusiasm after KC's impressive keynote lasted throughout the rest of the week. I had many great conversations with both OCaml and Haskell people about the new features and how exciting the future is for OCaml. In fact, the OCaml Farewell Reception, held at the Ljubljana Zoo, attracted three times the expected number of attendees because everyone wanted to keep talking about the new features coming soon in OCaml 5.0. I spoke at length with a senior Haskell programmer who was very interested in OCaml Multicore, and we all enjoyed sampling a local honey liqueur and having a hot meal, which helped warm us that cool, rainy evening. As a special treat, the ICFP organisers even arranged for a real live camel to make an appearance! It was a delightful evening, and the perfect ending to ICFP 2022.

In the end, the week in Ljubljana was fulfilling, both on a personal and professional level. After over two years of limited in-person events, it was truly refreshing to meet and network with colleagues face to face.

+ + + + + +The Famous Ljubljana Dragon Bridge

Tarides Sponsors High School Hackers

tarides — 2022-09-23T00:00:00-00:00

Tarides is excited to sponsor the Paradigm Conference (previously EsoLangConf) high school hackathon. This weekend, students from all over the world will team up to solve tricky programming problems, investigate diverse features of a range of programming languages, and build cool things!

At Tarides, we are always looking for new ways to increase the awareness and adoption of OCaml. The Paradigm Conference is a fantastic opportunity for all students interested in computer science to discover real-world use cases of OCaml.

If you are a high school student interested in using OCaml to solve fun and complex problems while learning new skills and meeting new people, you can register for the conference here.

What Makes this Conference Unique?

Created by Students for Students

Rohan Mehta and the organisational team provide a safe and inclusive hackathon space for attendees from anywhere in the world to explore different programming languages and concepts. Breaking out of the standard computer science curriculum, they have designed a hackathon specifically for high school students to showcase non-mainstream languages and the diversity of programming approaches available.

A Diversity of Programming Paradigms

The conference features different language tracks, focussing specifically on functional, array-based, and knowledge-based programming. Each team will choose from OCaml, Haskell, Clojure, Wolfram, and APL, letting them explore the unique features of these less well-known languages and discovering their benefits for themselves. The conference puts the joy of programming at the top of the priority list, giving students a fantastic opportunity to experiment and broaden their horizons.

If you want to learn about pattern matching, macros, or higher-order functions, then you’re in the right place!

Knowledge Sharing

The organisers have gathered learning resources for every language in the conference—a mammoth task in itself! Not only are they widely sharing knowledge that already exists (but may not be easy to find), but they are also creating new living documents for each language built from these existing resources and their own learning experiences.

Combined Coding Competitions and World-Class Lectures

The team has worked hard to create an engaging and interesting event by interspersing talks from industry language users and programming language experts with coding competitions and hackathon events. Naturally, any conference wouldn’t be complete without swag! Each attendee will receive a sticker and t-shirt, with additional prizes for the competition winners.

Get Involved!

Attending

Students will join teams (of up to 5) by either creating one in advance or registering as an individual. Everyone will be allocated to a team once the conference starts.

Mentoring

If you are a high school student and you’re interested in attending, you can sign up here. If you have a bit more experience with any of the languages featured, you can join as a mentor and help students grasp new languages and concepts.

Rohan and his team promise that if you attend Paradigm Conf 2022 “your programming worldview will be flipped upside down!”

You can find the Paradigm Conference on Instagram and twitter

Tarides Sponsors Girls Can Code

tarides — 2022-09-06T00:00:00-00:00

The tech industry has long struggled with a lack of diversity. This existing imbalance combined with social and educational problems such as early gender bias still tends to prevent lots of people, including women, from entering the field. Girls Can Code aims to help young women learn programming and gain valuable experience working on projects alongside other like minded individuals.

Tarides is proud to share that we sponsored a Girls Can Code summer camp! Between Aug 22nd and 27th, the camp offered participants a fully-packed week of programming, socialising, and learning.

What is Girls Can Code?

Girls Can Code is an initiative launched by the organisation Prologin, hosting summer camps specifically aimed at teaching young women about computer programming. No prior experience is required, and they accept participants from secondary school and up through the equivalent of A-Levels. Attendance is free, since the events are run by students who generously volunteer their time and expertise.

Camps come in two variants: long and short. The long camps last for a week and the short ones for a weekend. The short camps cover less content, but can be organised more frequently. They are always in the form of a practical introduction to computer science, but may focus on special topics. The week-long summer camps start with an introduction to Python and then continue with several tutorials on various topics. Finally, all participants have the chance to complete a personal project on either robotics, video games, or microcontrollers.

Impact

At Tarides, we’re committed to using our resources to foster diversity and inclusion. Our own goal is to have 50% of our tech roles be filled by women. For that to be a reality, not just at Tarides but everywhere, more women need to feel welcome in the tech space. Sadly, according to this Le Monde article, only 11% of French women chose to pursue IT careers in 2010. A more recent survey by Tech Nation showed that in the UK, only 26% of the tech workforce is made up by women. While this number is better, there’s still a lot of work to be done before women make up 50% of the workforce in tech.

Girls Can Code works proactively to inspire a generation of young French women and make a difference in the sector. They have been arranging summer camps since 2014 and have been growing steadily since. We’re proud to support their efforts for a more equitable future!

Tarides Goes on Holiday!

tarides — 2022-08-26T00:00:00-00:00

Relaxing in today’s world can be difficult. Taking the time you need to cool off, refocus, and explore something new requires a solid amount of time in which you can disconnect from daily habits and find a new beat.

At Tarides we address this by providing the framework needed for our employees to take that unbroken time away from work. In August, all Tarides employees get two weeks of paid leave to go away and come back refreshed.

How it Began

We first trialled the two weeks of leave in 2021 to help alleviate the stress caused by the pandemic. Taking a solid two weeks off would allow everyone to slow down and enjoy the last of the summer months (or winter for our Australian colleagues!) without having to worry about losing pay or using up annual leave in the face of an uncertain global situation.

The results were very positive, with a lot of good feedback across the teams. People came back refreshed and inspired, easily making up for the time away. An important takeaway from the Tarides team was that since everyone was away at the same time, no one had to worry about having a pile of work waiting for them when they came back. This made everyone’s holiday more enjoyable and restful.

It’s Back!

Since it was a very popular measure last year, we decided to reintroduce it as a recurring event! From August 8th to August 19th, Tarides had its official 2022 office closure. We hope the entire Tarides team took some time to go on adventures or simply relax before the rest of the year.

Taking time off does not just allow everyone to recharge their batteries, but it also lets them experience new things that can generate moments of inspiration. That said, rest is also very powerful: when we rest, we prepare for the challenges ahead, increase our resilience, and strengthen our resolve.

Here’s to a great rest of 2022!

Irmin in the Browser

tarides — 2022-08-02T00:00:00-00:00

Introduction

Over the past six months, I have been working on using Irmin in the browser, including irmin-server and the GraphQL interface. This has been fun and a great learning journey for me. Before this internship, irmin-server was primarily a Unix-based application. My project was to port irmin-server to work in the browser and design interfaces for people to interact with the store (Irmin stores).

I was paired to work with Patrick Ferris as my mentor and with the entire Irmin team, who all contributed immensely to this project.

Irmin and `irmin-server`

Irmin is simply a data store (database). It is based on the same design principle as Git with features to merge and branch data stores. Irmin has several stores (irmin-mem, irmin-indexeddb, irmin-fs, irmin-chunk, irmin-git) and store interfaces (irmin-http, irmin-graphql).

irmin-server is a high-performance server for Irmin. For efficient communication, it implements a specialised wire-protocol to send and receive data over a bytestream. It wraps an Irmin store, providing a way to connect to the server and access the store via its API using a client. But the client makes an assumption that the user is on a Unix machine, which makes irmin-server primarily a Unix-based application.

`irmin/irmin-client` in the Browser

In this modern age, it's become a necessity to make applications "offline first." Offline-First applications function without being affected by the intermittent lack of a network connection. It usually implies the ability to sync data between multiple devices. Irmin as a data store supports multiple backends, making it very portable. Plus, Irmin's mergeable replicated data-types make it much easier to build applications that can transform the state offline and resynchronise the state later, just like Git. With this concept, resynchronising Irmin stores (from server to client) is much simpler on irmin-server, which implements a specialised wire protocol for efficient communication. Making irmin/irmin-client work in the browser simply means that it would be possible to create offline-first web applications.

More information on offline-first applications can be found here

The Problems

An initial summary of the problem was published on this issue, but here is a quick breakdown of the problems we identified.

irmin-server was tightly coupled around conduit-lwt-unix: irmin-server was initially designed to be a Unix-based application that established communication with a client via conduit-lwt-unix. This became a problem because conduit-lwt-unix cannot establish a communication from a browser. This meant that there was a need to abstract the I/O module so that every client will provide its I/O.
Reuse some internal modules: We needed to reuse the irmin-server internal logic related to the protocol but provide a portable I/O interface that can work in the browser.
Provide a browser communication channel: We needed a non-blocking way to establish a channel to create communication between irmin-server and the browser, and also pass data across this channel.

The Solutions

`irmin-server` was tightly coupled around `conduit-lwt-unix`

Thanks to Zach Shipko, who abstracted the I/O library and split out irmin-client-unix and irmin-client-cli to have their own I/O module that depends on conduit-lwt-unix (here), a client can connect to a running irmin-server using its own I/O module. While he was working on the restructuring, I spent my time working on a sample project that combines dream with irmin-graphql (more on this project).

With the coupling out of the way, the next step was to create irmin-client-jsoo, a browser client with its own I/O module.

`irmin-server` was primarily a Unix-based application

The irmin-server initial architecture had to be restructured to accommodate other platforms. To achieve this, irmin-client was no longer coupled with a specific I/O implementation. Rather, a Unix-based one was provided over conduit flows, which are Lwt_io input and output channels. This channel was established over a TCP connection or a Unix domain socket.

Right now, irmin-server can communicate with two (2) clients: irmin-client-cli from a command line and irmin-client-unix from a Unix-based machine. This project was about creating a third client: irmin-client-jsoo, to be called from browser applications.

Enable communication from the browser

After considering other options to create a communication channel for irmin-client-jsoo, like HTTP, RPC, etc., Patrick suggested WebSocket, so we decided to go with WebSocket, a bidirectional communication protocol between client and server.

The Challenges

irmin-server uses flows to communicate between the server and the client and flows are bytestreams. WebSocket provides a bidirectional communication channel in the browser, but it is not stream-oriented rather it is message-oriented.

TCP (Transmission Control Protocol) is a type of protocol or standard to transfer information over the Internet while WebSocket is a message-oriented application protocol, which uses TCP as the transportation layer.

The idea behind the WebSocket protocol consists of reusing the established TCP connection between a client and server. Even though WebSocket is built on TCP, the data it passes is always either sent as a whole "message" or not at all. These implementations are non-blocking.

Since we are avoiding a full redesign of the irmin-server protocol, we had to make the message-oriented process seem like bytestreams of data.

More on `irmin-client-jsoo`

Communicating with irmin-server from the browser is very easy. You can achieve that by following these steps:

Pin irmin-server, using this command: opam pin add git+https://github.com/mirage/irmin-server/commit#013a28fd1507f8ba69494515533119804903aa99
Set up the server.

open Lwt.Syntax
+module Store = Irmin_mem.KV.Make (Irmin.Contents.String)
+module Server = Irmin_server.Make (Store)
+
+let main =
+  let uri = Uri.of_string "ws://localhost:9090/ws" in
+  let config = Irmin_git.config "penit" in
+  let* store = Store.Repo.v config in
+  let* main = Store.main store in
+  let* server = Server.v ~uri config in
+  let () = Format.printf "Listening on %a@." Uri.pp uri in
+  Server.serve server
+
+let () = Lwt_main.run main

Check out this implementation

Create the client and ping the server.

module Store = Irmin_mem.KV.Make (Irmin.Contents.String)
+module Client = Irmin_client_jsoo.Make (Store)
+
+let config = Irmin_client_jsoo.config (Uri.of_string "ws://localhost:9090/ws")
+let client = Client.Repo.v config in
+Client.ping client

More examples can be found on here

My Projects

Simple Mini GitHub: +I worked on this project to experiment with combining irmin-graphql with dream. This turned out simpler than I thought. You only need to expose irmin-graphql schema. In this application, you simply enter a GitHub repository, and the repository details such as name, date, author, commit message, and README file will be displayed. You can also open /graphiql and make queries.

The full code can be accessed here.

Pen-It-Down: +Pen-it-down is a note app that uses irmin-indexeddb and irmin-server to show an offline-first functionality. Users can type in their notes without being bothered about internet connectivity. You can create, edit, delete, and sync your notes to the server.

The full code can be accessed here.

Conclusion

Working on this project was challenging! I am so glad I had the opportunity to work on it, even though there were days I felt lost. Some days I was confused because it seemed I was doing the wrong thing. Other days I was happy because things worked as expected! It’s basically been about research and experimenting for me. I learned a lot from Patrick and Zach. I was exposed to networking concepts like the network layers, client-server handshake, data encryption, and decryption, and I got to try out WebSocket for the first time. I look forward to building more projects with OCaml.

Tarides is on the Wavestone Radar!

tarides — 2022-07-19T00:00:00-00:00

Cybersecurity is a growing concern for individuals and companies alike. At Tarides, security is at the centre of every solution we provide, and this year we have been recognised for our efforts! We’ve been accepted to Cyber@StationF’s acceleration program and are now featured in the 2022 Cybersecurity Startup Radar.

Tarides is proud to announce that we’re part of the 2022 Cybersecurity Startup Radar by Wavestone and BPIFrance! It spotlights promising startups who are making a difference in the cybersecurity arena. Tarides has been featured in the Radar twice before, in 2019 and in 2020.

Tarides is featured under the ‘IoT Security’ section of the Wavestone Radar, demonstrating our commitment to creating safer and more efficient software for the IoT (Internet of Things) ecosystem. At Tarides, security-by-design is at the core of everything we do; it’s our guiding development principle that we use to produce a variety of solutions addressing the different challenges present in this space.

Tarides’s Cybersecurity Solutions

Most of today’s software solutions are very complex and as a result often inefficient and vulnerable to attack. It is well documented that IoT technology and devices can pose safety risks due to their development environment and limited processing capabilities that don’t offer full protection from attacks.

Our technology solves these problems in a revolutionary way by combining the features of OCaml (focused on security, safety, and efficiency) with state-of-the-art tools (as part of MirageOS) to provide solutions to complex problems where mistakes can have disastrous consequences.

Firstly, we provide several solutions relating to device connectivity, from internet protocols to low-bandwidth networks. Secondly, we build custom applications that are securely deployed on IoT devices either on bare-metal or within hypervisors. MirageOS provides an efficient IoT environment with a small footprint. Finally, we address the IoT security layer via formally verified cryptographic libraries and other security building blocks that match the latest standards.

We maintain a special focus on cybersecurity to ensure that efficiency and security go hand-in-hand, and that one is not achieved at the expense of the other. We develop and maintain secure, fast, and performant code solutions that leverage OCaml's strong safety features for reliable results. We collaborate closely with the thriving open-source community surrounding the language, which constantly tests and audits its performance.

About the Radar

The Radar is used to analyse the French cybersecurity sector: what solutions are being worked on, what trends are emerging, and what kind of innovation is happening. In light of the current geopolitical climate, cybersecurity is a major strategic issue now more than ever. Consequently, this year’s Radar is perhaps even more salient than those of previous years, as it highlights the role of France’s innovation ecosystem in advancing cybersecurity goals. +By being featured, Tarides gains a lot of visibility in the sector and can also discover other startups and promising projects in the same field. We’re excited to be part of this great networking opportunity!

About Wavestone & BPI France

Wavestone operates at the intersection of management and consulting, helping their clients not just overcome but master challenges whether they be digital, competitive, or environmental. Their mission is to guide organisations during critical transformations, helping them obtain the best results. Furthermore, they are also committed to promoting ethical and sustainable solutions that benefit society as a whole.

BPI France is a financial institution whose mission is to support entrepreneurs and visionaries who take risks to achieve their goals and grow their businesses. BPI France has many resources that they make available to companies, offering support for innovation, coaching, acceleration programmes, and international expansion. They focus on smaller businesses such as microbusinesses, SMEs, and mid-caps, but also offer solutions for larger companies.

Faster Incremental Builds with Dune 3

tarides — 2022-07-12T00:00:00-00:00

In February 2022, we released Dune 3.0. This updated version is the result of considerable development work over the previous six months. Dune 3.0 contains many new features, one of which is “watch mode,” an exciting new feature explained below.

As a build system, Dune’s main goal is to build targets. These targets can be either files (like an executable file) or “aliases,” a group of targets that can have a visible outcome (like running tests). By default, when running a build, Dune receives a target. Dune will then build it and exit. For example dune build (an alias for dune build @all) will build everything it knows about, then exit.

When working on a piece of code, many developers use an edit-save-build loop:

Edit a piece of code
Save the corresponding file
Run a build command

Using the outcome of the build (i.e., Did the build work? Did the tests all pass?), developers start a new iteration of this loop manually, but it’s more efficient to have a quick, automated iteration process. This is the goal of the “watch mode.”

When active, Dune will watch the source files in a project, and when one of them has changed, it will re-execute the same build command automatically and display the results of the build. It doesn’t exit automatically, so it continues watching for changes. This is more efficient because the developer can stay focussed in their text editor and see the build start automatically when the file is saved. You can enable watch mode by passing the -w flag to dune, like dune build -w.

A simple implementation is to have a special process check for file changes in the source tree and run the build command when something has changed. This works, but it isn’t a very precise solution. First because the external process doesn’t know about the relationships between the files, so it will run more builds than necessary. For example, changing a README file usually should not trigger a new build because it isn’t a source file. But also, there are various subtleties to handle. If a file is changed while a build is running, a new build should be started, but the previous one should also be cancelled.

For these reasons, it’s more efficient to have the build system itself “drive” the watch mode. This is how it’s implemented in Dune 1.x and Dune 2.x. When starting a build in watch mode for a certain target, Dune computes the set of files that can influence this target (using the build rules) and calls an external process that can subscribe to file changes. When a file changes, Dune cancels existing builds and will start a new one.

This is better, but it’s still not very efficient. To see why, let’s see what Dune does and how it can be fast.

To run a build, Dune needs to do two things:

Load the rules: detect the workspace (determine which files to consider), parse the dune files (open them, transform them into s-expressions and stanzas), and interpret them (execute the logic to transform the stanzas into rules)
Execute the rules: copy files around, call external processes, etc.

The time it takes to load the rules is related to the size of the current workspace (number and size of dune files). This is particularly noticeable in organisations that use a monorepo (all the source code in a large Dune workspace). It's difficult to make this step fast because it has a lot of work to do, but it's doable by avoiding computing the same things over and over, made possible by an internal memoisation framework. An initial version of this system is described in this blog post.

The time it takes to execute the rules depends on the amount of work necessary. For example, a clean build needs to execute most of the build actions, while a second full build usually needs to execute no rule at all. To make this step fast, Dune tries to avoid executing actions that wouldn't change the final outcome (a technique called early cutoff), and it executes independent actions in parallel.

In the context of the watch mode, whenever a new build starts, Dune has to forget everything it knows about the workspace, so it will reload all the rules. This is pretty wasteful.

To do better, the new watch mode in Dune 3 makes rule loading incremental. For example, if a dune file is edited to add a stanza, Dune parses it again, only adding the new rule. The other ones are not interpreted. This ensures very fast iteration times.

This project was challenging because for it to work, everything in the Dune core had to be ported to the memoisation API. For instance, the library loading code (which looks for library definitions in the current opam switch and in the Dune workspace) relied on a “classic” cache (a global hash table) to avoid parsing files repeatedly. However, this does not play nice with the memoisation API, which assumes that the functions it caches are all pure. So, in Dune 3, this piece of code has been rewritten on top of the memoisation API. This has another benefit: since file system accesses (“does this file exist?”) are cached too, the memoisation API now has an idea of which functions can read which files. This is used to re-evaluate only the affected parts of the rule graph once a file is modified.

Thanks to that work, watch mode is now a lot more responsive than in Dune 2.x. This performance improvement is barely noticeable in small-to-medium-sized projects, but it is essential in a workspace with several million lines of OCaml code. In such a setting, re-evaluating rules over and over (either by manually running dune build or by using the strategy in Dune 2.x) means that the feedback loop takes dozens of seconds instead being almost instantaneous.

As Dune performance improves, it's able to support workspaces that are larger and larger. This means that the bottlenecks shift to different places. We'll continue to improve Dune so that it stays a build system that's convenient to use and endlessly scalable.

The Magic of Merlin

tarides — 2022-07-05T00:00:00-00:00

Tarides provides support and development services for OCaml tools, packages, and libraries for our commercial partners and for the benefit of the entire OCaml community. We focus on groundbreaking innovation, feature development, and crucial maintenance of OCaml-based projects. One of these projects is called Merlin, an advanced Integrated Development Environment (IDE).

Overview

When someone hears the word "Merlin," images of King Arthur, the Round Table, and the Holy Grail might come to mind. This fantasy world introduces us to Merlin, the mighty wizard who becomes Arthur's advisor and mentor. When speaking in the world of technology, our Merlin's magic comes in the form of a powerful editor service that offers completion, typing, navigation, refactoring, and code generation. This IDE was made specifically for OCaml, so it complements the safety and expressiveness of the OCaml language with powerful tools. In short, Merlin is a loyal companion that magically helps OCaml developers be more productive and write better programs!

Installing Merlin

Merlin integrates with most editors, including Visual Studio Code (VSCode), via the Language Server Protocol (LSP). It also implements custom features on top of LSP to enable powerful developer workflows that are only available in OCaml. As a result, Merlin helps developers write in OCaml more easily, as they’re provided with instant feedback on any possible errors that they could make. It helps train these programmers to make fewer errors in the future and eases project maintenance by automating complex (and otherwise error-prone) workflows.

The easiest way to install Merlin is by using VSCode's OCaml extension. See the manual for more information. You can also use Merlin through Vim or GNU Emacs. Read more on OCaml.org's Merlin page.

Once the installation process is complete, your editor will automatically start Merlin whenever an .ml or .mli file is opened.

Voilà! So easy!

Merlin in Use

Here's a glimpse of what Merlin looks like in VSCode:

+ + + + +

Merlin is the default IDE tool for OCaml developers. It's utilised by many commercial OCaml users who are funding maintenance work and evolutions of the project. For instance, Jane Street developers, sysadmin, and traders confidently use Merlin to browse, maintain, and modify an OCaml codebase that runs into millions of lines of code. This codebase provides the foundation for Jane Street's financial market trading around the world. It critically depends on tools (such as Merlin’s) to ensure it can continue to evolve while functioning safely.

Beneficial Features in Merlin

One of Merlin's main developers, Frédéric Bour, says his favourite feature is "completion," which has the ability to complete a prefix typed by a programmer in a manner that is (somewhat) relevant in the context. He says, "I like it for two reasons: less things to remember (and in a programming language, usually you have to recall exactly because there is not much room for fuzzy interpretation) and also it makes things 'discoverable.' Sometimes you work in an area that is new to you, and looking at the Merlin view with its suggestions really helps engineers become acquainted with this program."

The Tarides CTO, Thomas Gazagnaire, loves that Merlin "is the perfect companion to any professional OCaml developer. It helps navigate completely new codebases by providing the necessary feedback to learn the project (and the OCaml language!) quicker. It is also super useful to refactor large pieces of existing code, with immediate feedback and hints. I remember very clearly when I started using Merlin on my projects. It provided me a great productivity boost that completely changed the way I programmed in OCaml and made me much more effective. Tarides is now committed to make sure this tool continues to be supported actively. We're always adding new features to improve developer productivity, like value and type renaming, semantic search over a whole project, etc.”

Learn More

If you'd like to read more about Merlin, or become a contributor, visit its GitHub repo, and feel free to open an Issue if you have any suggestions. Please contact us if you would like to subscribe to commercial support or discuss future development.

Thales Cyber@Station F Selection

tarides — 2022-06-28T00:00:00-00:00

The online world is becoming an increasingly bigger part of our everyday lives, bringing the issue of cybersecurity to the forefront of more and more minds. At Tarides we put security at the centre of everything we do, and we’re honoured to be part of Cyber@Station F in 2022.

Tarides is thrilled to announce that we have been selected for the Cyber@Station F Acceleration Program! It’s a fantastic opportunity for Tarides to exchange information and collaborate with other startups, as well as connect with the cybersecurity giant Thales.

The Program

The Cyber@Station F program was established by Thales Digital Factory at Station F to “help accelerate startups’ development in cybersecurity by providing them advice, expertise, and access to our big markets.”

Thales Cyber@Station F is a startup acceleration program that centres around the areas of cybersecurity & trust. It guides its participants through four stages: Select, Define, Deliver & Test, and Business Acceleration. The stages are designed to give each startup an opportunity to come up with and test a proof of concept with potential clients, which if successful is then developed further.

What We Do

At Tarides we know that a lot of today’s technology solutions are overly complex, vulnerable to attack, and time consuming to develop. As an alternative to this, we provide secure, safe, and efficient solutions by leveraging the features of OCaml in combination with cutting-edge tools and technologies. In collaboration with a rich open-source community, we develop and maintain the OCaml language, the operating system MirageOS including Unikernels, as well as a range of developer tools.

The OCaml language is efficient, writes safe and secure code, and is easy to maintain and adapt thanks to its modular nature. When combined with MirageOS and Unikernels, which reduce runtime complexity for lightweight and accurate results, OCaml’s already impressive features are used more effectively. Finally, our range of modern, easy-to-use tools offer developers a range of options for writing projects in OCaml.

Cybersecurity is at the heart of what we do, and we’re excited to combine our knowledge and experience with the substantial resources offered by Thales and Station F. We are also looking forward to engaging with other companies in the same industry and discovering new opportunities for our projects and our clients.

Thales

Thales is a global leader in technology innovations and solutions specialising in digital and ‘deep tech’ innovations such as Big Data, artificial intelligence, connectivity, cybersecurity, and quantum technology. Their goal is to invest in these technologies to build a better future that people can trust. Thales focuses on five vertical markets: digital identity and security, defence and security, aerospace, space, and transport. Their clients play a central role in these socially important markets.

Station F

Located in Paris, Station F is a startup campus that hosts over 1000 startups. StationF offers everything an entrepreneur needs to start and grow their business, making over 35 public services, 150 perks, and 600 workshops and events available to people in their network. Their services include startup programs on specific themes, special mentorship offices, a Flatmates service, exclusive discounts, and much more.

More Coming Soon

The programme started at the beginning of June, and we’ll follow up with more developments as they happen. In the meantime, we’ve selected some relevant posts and information on our work in cybersecurity you can read if you want to know more. We’ve received funding from the EU for our work on a secure open messaging platform, we’ve been laureates of the i-Lab innovation contest for our work on Osmose, and we’ve won the 2020 FIC startup award. If you want to read up on the technical side of things, these papers on unikernels and on Osmose are a good place to start.

Team Tarides Visits a 17th Century Chateau

tarides — 2022-06-23T00:00:00-00:00

Everyone at Tarides recently had the opportunity to meet up in person for the first time! Since the global pandemic left much of our distributed team unable to meet, we organised a working retreat that brought all our teams together to work, learn, and have fun.

This was the first formal retreat we’ve held as a company, and our goals were to provide an opportunity to disconnect from day-to-day work, meet with new people, and discuss new projects - all in a novel environment surrounded by fresh air and open space. During the course of the pandemic, Tarides grew from 36 to 71 people based in 11+ countries, with few of them ever having met in person. As soon as it was safe to do so, we knew that gathering together would give us an invaluable experience in building and reinforcing our own company culture.

This May, all members of Tarides were invited to a beautiful 17th century chateau at Les Prés D’Ecoublay, surrounded by fields, orchards, and lush green forests. Attendees participated in exciting workshops, team-building activities, tech-talks, and inspiring discussions. Over the course of two days, everyone had plenty of time to collaborate, socialise, and eat fantastic food!

What We Got Up To

The fun began at 9am on Thursday, May 19th, when Tarides’s distributed global team gathered to make their way to what was to be their castle for the next two days. Greeted by a feast of French pastries and fresh fruit, introductions were made between people from Australia, France, India, Germany, the USA, the UK, and many more countries.

Tech Talks & Tutorials

Over the next couple of days, the itinerary left plenty of space for knowledge sharing via tutorials and ‘tech talks’ or presentations. KC Sivaramakrishnan gave everyone a sneak peak of OCaml 5 with a detailed tutorial. The tutorial is openly available on GitHub. Please give us feedback if you try it!

Engineers Sonja Heinze and Jan Midtgaard held presentations (so-called ‘tech talks’) in the main hall for everyone’s benefit. Sonja’s talk was on the benefits of Outreachy, an open-source internship coordinator that creates opportunities for those most affected by underrepresentation or discrimination. Jan introduced everyone to property-based testing, a fascinating way to test the ‘correctness’ of code. The goal of the 'tech talks' is to provide a safe space for people to share their work with others, where everyone is welcome to ask questions and discuss the topics covered.

Cross-Team Collaboration

There were also plenty of opportunities for teams to meet and work together, with individual meeting rooms readily at hand. The coffee machines (and accompanying sweets) in each room significantly boosted the productivity of all teams! Teamwork was not limited to individual groups, but time was made for cross-team brainstorming and collaboration.

All of the teams at Tarides are working towards the big OCaml 5.0 release scheduled for later this year, and we took the opportunity to use this release as a focal point to align our goals for the next few months. Each team spent some time together to conceptualise their team-specific goals first, and then the Team Leads followed up with an "office hours" drop in session to discuss cross-project interaction and ideas. Creating dedicated space to think beyond the usual daily tasks and projects proved to be really valuable, producing insights and connections that may have otherwise been missed.

Fun & Games!

It wouldn’t be a working retreat without opportunities for relaxation and fun! We got competitive in a treasure hunt that had us searching the forest and tall grasses for clues. There was plenty of time to explore the vast castle grounds, which included a pool, archery arena, ping pong table, karaoke pavilion, and much more.

Until Next Time

The off-site was a huge success! As a distributed team, it’s important to occasionally get together and put a face to a name (or Slack and GitHub handle!). Everyone at Tarides looks forward to the next off-site, wherever it may take place.

Functional Conf 2022

tarides — 2022-06-21T00:00:00-00:00

This year, Tarides attended the 2022 Functional Conf in India. Tarides’s engineers Sudha Parimala and Shakthi Kannan gave presentations on the OCaml platform and Sandmark, a continuous benchmarking tool for Multicore OCaml.

The Functional Conf is a three-day conference on everything functional programming! It’s a great event for beginners and experienced developers alike. Beginners have the opportunity to be introduced to different functional programming languages and understand their fundamental principles, and those who are more experienced have plenty to learn from both participants and speakers on how they have leveraged functional programming in their projects.

This year’s Functional Conf was held online and attended by people from around the world. It has been referred to as “Asia’s premiere functional programming conference” and welcomes participants from a broad range of backgrounds.

OCaml Platform 2022

Sudha Parimala’s talk is a great introduction to OCaml and its features, covering the installation process and "Hello World," as well as more advanced topics such as the text editor, publishing a library, and debugging. For the Functional Conf, it was a great way to give people interested in functional programming a taste of OCaml and what makes it stand out.

The presentation is a fantastic resource for people who are starting their journey in OCaml and want to know more about what they can do with the language, as well as for people further ahead looking for inspiration on different ways to progress.

Benchmarking (Multicore) OCaml

Shakthi Kannan’s talk centres around Sandmark, the benchmarking suite designed to test various parts of the OCaml compiler and its runtime. Benchmarking is a challenging process. As a result, there are few tools available that do the job well. OCaml’s Sandmark can test various performance axes such as CPU, memory, and I/O, as its tools build the compiler under various configuration settings. It also comes with a dashboard that lets the user explore the results of benchmarking runs in an interactive and easily digestible format.

In his talk, Shakthi describes the journey to Sandmark, originally developed to support the Multicore OCaml project. He covers the challenges the team faced and the lessons they learned along the way, especially with an evolving programming language and the need to support multiple CPU architectures. It’s an amazing resource for teams who are looking to set up their own benchmarking procedures, and it is also a great example of how to approach a difficult task as a team.

In Conclusion

The Functional Conf is a great conference that brings the growing community of functional programmers together. It offers opportunities for people to learn about functional programming and exchange information with others in similar fields. Tarides is proud to have participated in their effort to bring functional languages to the forefront of programming.

To learn more about Functional Conf you can visit their website, along with the individual pages on Sudha’s talk and Shakthi’s talk, respectively.

OCaml 5 Alpha Release

tarides — 2022-06-15T00:00:00-00:00

OCaml 5 is live! This major release introduces domains and effects, delivering unprecedented speed and efficiency to OCaml. Testing shows that OCaml 5 is able to outperform Go and closely match Rust in terms of performance. Keep reading for more details!

Tarides is thrilled that the alpha release of the long-awaited OCaml 5 is live! OCaml 5 is the culmination of over 8 years of research and engineering into concurrency and parallelism support for OCaml, made real thanks to hard work and dedication from all corners of the community. Tarides has been a major contributor to the engineering effort. Our engineers have contributed not only to the core compiler but also to the tools around release readiness, ecosystem compatibility testing, and continuous performance monitoring.

If you are using OCaml in an industrial setting (or if you are interested to do so), we'd like to make sure everything is ready for you to move to OCaml 5 and benefit from the new performance boost. Tell us what you need in this user survey.

What Kind of Performance to Expect?

This update brings unprecedented results in terms of performance, with an HTTP server based on OCaml 5’s Eio being able to serve 1M+ requests/sec, outperforming Go’s nethttp, and closely matching Rust’s hyper performance! This is just a small indication of OCaml 5’s potential in terms of speed and efficiency.

+ + + + + + + + + + +

As it’s an alpha version, it’s still subject to some change and fine tuning. This means that we need your help to make OCaml 5 better! Please give plenty of feedback and report any bugs you find.

Under the Hood

The alpha release of OCaml 5 adds support for shared memory parallel execution via domains and a new model for concurrent execution via effect handlers.

Domains enable shared-memory parallel programming that allow OCaml programs to run on multiple cores. With domains, OCaml programs will scale better by exploiting multicore processing. Effect handlers are a mechanism for concurrent programming. With the introduction of effect handlers, simple direct-style OCaml code will be flexible, easy to develop, debug, and maintain. No more monads for concurrency! These features will benefit the entire ecosystem and community, and we expect it to attract many new users to the language.

The Standard Library is gaining several of the parallelism primitives previously only found in the Threads library (Condition, Mutex, and Semaphore). Interestingly, having added domains and effects, we hope and expect that most users will never need to use them directly! Instead, we warmly encourage users to look at adopting domainslib to parallelise programs and Eio as a replacement for Lwt/Async monadic-style concurrency.

This work seeks to remain entirely backwards-compatible. Programs written for any version of OCaml 4, even if they use the Thread library, will continue to work with the same semantics, similar performance, and as always for OCaml, without crashes.

Next Steps

Installation

For instructions on how to install OCaml 5 on your machine, Florian Angeletti’s Discuss post goes into great detail on how to do so, depending on what version of OCaml you’re running and what machine you have. KC Sivaramakrishnan has also created a tutorial on OCaml 5 that introduces its new parallelism features, a great resource for anyone looking to make the most of the update.

Other OCaml 5 documentation includes information on parallelism, effect handlers, and the memory model.

Feedback

We want to reiterate that as with any alpha release of OCaml, we’re keen to hear about bugs and performance regressions. The move to parallel OCaml may bring new debugging challenges, but it remains the case that pure OCaml programs which do not use unsafe features should absolutely never crash. We’ll be taking part in the discussion on GitHub, Discuss, and Twitter.

The change in major version number (from 4.n to 5.n) may result in minor breaking changes which affect your packages, particularly if you’ve been allowing some deprecation warnings to slip through in the past! We’ll be following up with more information about the required tweaks that may be required for packages supporting both old and new versions of OCaml, as well as with specifics on the testing infrastructure.

Survey

As discussed above, we’ve created a user survey to help us get a better sense of how people are planning on using OCaml 5. It would be very helpful if you could fill it out for us, which should only take a few minutes.

OCaml 5 Timeline:

The beta release will take place once these issues have been resolved.
The final release is expected in September.
There is no time limit on reporting bugs, so please report them here.

Adding Merkle Proofs to Tezos

tarides — 2022-06-13T00:00:00-00:00

The Upcoming Tezos Jakarta Protocol will support compact Merkle +proofs to scale the network's trust infrastructure. +This allows nodes that do not trust each other to agree on the +validity of Tezos transactions with orders of magnitude smaller +storage requirements. +For instance, the block 2,400,319, +containing 402 transactions and 638 operations, +can be validated using a Merkle proof of 6.3 MB instead of requiring +a Tezos node with at least 3.4 GB of storage, a savings of 99.8%!

Tarides contributed to Jakarta by extending the Tezos +storage system to support compact +storage proofs. This +feature extends the compact cryptographic representation of the ledger +state to sequences of operations. As a result, nodes that do not trust +each other can still agree on the series of operations' validity, +even if they don't know the entire contents of the ledger. The +upcoming stateless nodes (like Tezos +light-client), +proxies, and mechanisms that allow the exchange of trust between disjointed tamper-proof storage (like L2 +transactional-rollups, +L2 smart-contract rollups,...) will use these proofs to scale the +Tezos trust infrastructure to new heights.

The Merkle Proof API is one of the last major features that we integrated from +Plebeia. It is +the result of a years-long collaboration between +DaiLambda and Tarides to improve the +storage system of Tezos.

A (Very) Quick Tour of Tezos

The Tezos network builds trust between its nodes by using two components:

(i) a tamper-proof database that can generate cryptographic hashes, +which uniquely and compactly represent the state of its contents; and
(ii) a consensus algorithm to share these cryptographic hashes +across the network of (potentially adversarial) nodes.

Both components have seen impressive improved performance +recently. First, for (i), we've discussed +the improvements that we released in Octez v13 to improve +the efficiency of the storage component by a factor of 6. Second, for (ii), the consensus algorithm in Ithaca 2 changed from a +Nakamoto-style algorithm (like Bitcoin) to +Tenderbake -- a Byzantine Fault +Tolerance consensus with deterministic finality. This change +significantly improved the time it takes for converging towards a +uniquely agreeing state hash in the Tezos network.

The Merkle proofs that we +introduced in the Jakarta Protocol will allow us to improve the +chain's performance even +more.

The Tezos Ledger is a Merkle Tree

Tezos represents the ledger state (for instance, the amount of tokens +owned by everyone) as a Merkle tree, using the +Irmin storage library. Merkle trees are immutable +tree-like data structures where each leaf is labelled with a +cryptographic hash. Each node's hash is then obtained by +recursively hashing its children's label. Tezos then combines that +root hash computation with its consensus protocol to make sure every +node in the network agrees on the ledger's state.

But there is another interesting aspect of Merkle trees that was not +exposed and used by Tezos until now: Merkle proofs. In the protocol +J proposal, we are introducing a new feature: Merkle proofs for +Tezos as partial, compressed, Merkle trees. In a blockchain, as in +Tezos, Merkle proofs are an efficient way to verify the integrity of +operations over Merkle trees. For this reason, Merkle proofs are +a central part of the optimistic rollups projects that will +be available with the Tezos +Jakarta Protocol.

In collaboration with DaiLambda, Marigold, Nomadic Labs, and TriliTech, +we have integrated Plebeia +Merkle proofs in Irmin. Plebeia use Patricia binary trees that are capable of +generating very compact Merkle proofs. For instance, the proof of +100 operations can be represented within 46 kB, while storing the full +Tezos context requires 3.4 GB of disk storage to store the relevant context. +This compactness comes from its specialised store structure and clever +optimisations, such as path compression and inlining. We have been +working with the DaiLambda team to unite Irmin and Plebeia's strengths +and bring built-in compact Merkle proof support to Tezos. We added +support for both the existing storage stack, where trees have a +branching factor of 32, and for new L2 storage systems that could use +binary trees directly. We have also worked with Marigold and Nomadic +Labs to propose an alternative representation of these proofs using +streams, that comes with a simplified verification algorithm. +A stream proof encodes the same information as a regular proof. +However, instead of being +encoded as a tree, the proof is encoded as a sequence of steps that +reveal a Merkle tree lazily, from root to leaves.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Kind of Proofs	1 op.	100 ops.	1k ops.	10k ops.
binary Merkle trees	0.7kB	46kB	371kB	2.8MB
stream binary Merkle trees	1kB	75kB	602kB	4.5MB
Merkle B-trees (32 children)	3.1kB	158kB	1232kB	7.8MB
stream Merkle B-trees (32 children)	3.1kB	158kB	1238kB	7.9MB

+
The table above shows the size for such proofs, using 2.5M entries (the current number of entries in /data/contracts/index in the Tezos context). We are simulating 1, 100, 1000, and 10_000 random read operations on the entries, and we display the size of the related proofs.
+

An Example

Let's look at a simple example of a Merkle proof produced for ensuring +tamper-proof banking account statements.

We can model a bank that stores its customer balances in the form of a +Merkle tree. To avoid publishing the entire contents of its customer +accounts, this bank can publicly export the bank's Merkle tree's +hash. To let 3rd-parties validate an operation, it can also produce +Merkle proofs that reveal the balance of some customers. Anyone in +possession of a Merkle proof can hash it and verify that it hashes +identically to the public hash announced by the bank. This equality of +hash is proof of correctness.

Our bank contains the balances for Eve (30 coins), Ben (10 coins), and +Bob (20 coins). It stores the customers in a radix tree (Eve's balance +is stored under "e", "v", "e").

Irmin stores data as hash trees: whenever data is added to the +database, the corresponding nodes in the tree are hashed in order to +then generate the hash of the root node. The hash of a commit also +acts as a reference for accessing it in the future. This storage +format is close to Merkle proofs. It suffices to blind the part of the +tree that a transaction has not accessed to generate its proof. For +example, the account statement for Eve is in green. It doesn't leak +sensitive information about the other customers: only the letter "b" +is leaked. The hash corresponds to Bob and Ben's subtree.

+ + + + +

+
The Figure shows the difference between a Merkle tree (on the left) and a Merkle proof (on the right). In Merkle proofs, some subtrees can be blinded and represented only by their hash. Here the subtree under H2 is blinded and replaced by its hash. Thanks to this, Merkle trees and Merkle proofs will have the same root hash. Merkle proofs are hence very useful to provide Merkle tree summaries for a subset of the full data available.
+

Merkle proofs are thus partial Merkle trees, with the same root hash. But proofs +can also be represented using an alternate definition: a stream of elements +that needs to be visited in order to build the tree's root hash. There is +a one-to-one correspondence between the two representations, but stream proofs +are easier to implement as they encode the order in which nodes have to be visited to verify the proof. However, stream proofs need to carry the hash of all the intermediate nodes, while tree proofs can omit those. As a consequence, tree proofs are smaller than stream proofs, as shown in the above table.

For instance, in the above Figure, the equivalent stream proof is the sequence:

A leaf with 30 coins;
A node with hash H3 with a child ("e", "30 coins");
A node with hash H1 with a child ("v", H3);
A node with hash H0 with two children ("e", H1) and ("b", H2).

A verifier can just apply these elements in sequence to verify that H0 is +a valid root hash.

Merkle Proofs in Irmin

If you want more details about the Merkle proof implementation, head +over to the +documentation +or at https://github.com/mirage/irmin/pull/1802 for the example above +revisited in Irmin.

OCaml Matrix: A Virtual World

tarides — 2022-06-09T00:00:00-00:00

Introduction

One of Tarides's projects is to create an open and secure infrastructure for communication protocols, initially focusing on emails and Matrix. This will allow organisations to self-host their messaging services, using either personal cloud resources or low-cost embedded devices. Individuals and organisations can use this framework to avoid having their emails and messages read and managed by third parties.

Every component of our system is carefully designed as independent libraries, using modern development techniques to avoid the common reported threats and flaws. For instance, the protocols' implementation is written in a type-safe language and tested with state-of-the-art, coverage-driven tests, such as fuzzing. Then it's deployed as unikernels for enhanced security, model quality, and library portability. The combination of these techniques will increase users’ trust to migrate their personal data to these new secure services.

The Matrix

When hearing the word Matrix, people invariably think about Neo and his ability to see the code behind his virtual world. In lieu of the cultural connection to the popular film series, the Matrix Communication Standard creators respond to the implicit assumption regarding their choice of name: “We are called Matrix because we provide a structure in which all communication can be matrixed together.”

Communication is essential to our society to both create and maintain relationships, whether personal or professional. As we progress further into this age of information, people communicate and stay connected through online and text-based communication. Gone are the days when someone would pick up a phone to call a friend or family member. Now, most people send a text message or email as the default. Thus, online communication has become the norm in our current society. Inevitably, this online communication is vulnerable to malicious actors trying to invade our privacy and hijack our correspondence. Tarides has addressed this issue and aims to host community discussions about open-source projects.

Matrix is an established protocol for human-to-human and human-to-machine communications, including instant messaging. OCaml Matrix is an OCaml implementation of the Matrix protocol. This provides a secure communication layer which is based on MirageOS’s unikernel technology in order to reduce the attack surface. It uses Irmin as storage for the communication content to ensure integrity, and we have integrated it into the CI system for all OCaml projects.

Let's take a closer look at the ocaml-matrix component and explore some details about the Matrix Communication Standard to see if it’s indeed communication matrixed together or if it’s comparable to Neo’s Matrix with people plugged into a virtual world.

Matrix Beginnings (History)

Matrix is an open standard for interoperable, decentralised, real-time communication over the Internet, created in 2014 inside Amdocs, a company specialised in software and services for communications. Matrix provides fully decentralised and federated architecture, so they don’t store users’ information in a centralised location. This means when people join one of the Matrix virtual rooms to send messages, video chat, or share files, their exchanges are truly private, especially with Matrix’s end-to-end encryption. Matrix’s decentralised, federated architecture ensures communication integrity and availability in every room.

Matrix is openly specified and implemented with the open-source reference implementation server Synapse and client Element, previously Riot, which already have several, astute security features and allow end-to-end encryption. Starting in 2018, the French Government deployed a private federation of Matrix home servers and Tchap, an open-source client forked from Riot. The French National Cybersecurity Agency (ANSSI) jointly works with the Interdepartmental Digital Directorate (DINUM) on a cybersecurity audit of Tchap. Matrix’s interesting security features include end-to-end capable search and enables private rooms’ end-to-end encryption by default.

Matrix Reloaded (Architecture)

Users interact by sending and receiving events in Matrix rooms. Each Matrix user registers a homeserver that is identified by a unique ID, like “Neo:tarides.com.” The registration goes through a client application that connects to a Matrix homeserver via the client-server API. This allows users to perform actions such as sending messages, controlling rooms, or synchronising their conversation history. All communication in a Matrix room replicates across the room participants’ homeservers, so every homeserver connected to a room stores the content of the room’s history.

Basically, the user communicates to a home server via a client application. Once the user decides to join a room, the client sends this request to the homeserver, and it’s the homeserver’s responsibility to connect the user to the room, to store the history of the messages of that room, and to send the messages back to the user. The homeserver gets all this information by talking with the other users’ homeservers in that room. This way, if a homeserver goes down, the conversation can continue as the remaining homeservers are still exchanging messages. When a homeserver comes back online, it resynchronises the messages. It receives old ones from other homeservers and inserts its own into others’ timelines.

+ + + + +

Matrix Architecture Image Description: Matrix users communicate via Matrix clients, which can be web client, a mobile client, desktop clients, or embedded clients built into existing apps like Slack via Matrix bridges. It could even be a piece of hardware (e.g., a drone) that is Matrix enabled. A user's client connects via an unique ID to a single homeserver, which stores the communication history and account information for that user. It also shares data with the wider Matrix federation by synchronising communication history with other homeservers. The conversations among users take place in rooms that have their contents replicated across all of the homeservers associated with the users present in a room.

The centralised communication architectures keep the data within their own systems, which induces a series of security issues. For example, usually the centralised systems offer very little transparency regarding their implementations. This means that, for the claimed purpose of security, the centralised system could either hide backdoors or have security flaws that pose serious issues to privacy. By contrast, an open-source system promotes transparent development, which provides assurance regarding the liability of the implementation by allowing ad-hoc code audits. Moreover, the decentralised architecture empowers users to host their own conversations rather than all their data being stored by the service provider. This renders less incentives for attacks targeting massive data leaks and, in combination with the confidentiality ensured by the end-to-end encryption, induces an increased level of security while promoting ownership and data sovereignty.

Matrix Revolutions (in OCaml)

Matrix’s Hall of Fame shows several ethical researchers’ investigative work into Matrix’s security vulnerabilities. For example, a recently discovered buffer overflow produces a considerable information disclosure in other Matrix implementations, such as Element. At Tarides, we mitigate a consistent class of these vulnerabilities with the OCaml development environment, which provides secure-by-design guarantees for the OCaml Matrix project.

+ + + + +

OCaml Matrix Architecture: The OCaml CI Client is a bot that communicates with Matrix servers via the TLS protocol, such as the ocaml-matrix server. The ocaml-matrix server is the unikernel that ensures the communication with other Matrix servers from the federation to synchronise upon events in the Matrix rooms. For this purpose, OCaml Matrix exchanges DNS information with a unikernel that plays the role of a Primary DNS Server and connects with an Irmin storage unit to save the rooms’ states.

Our ocaml-matrix server manages its own clients, who create public rooms for events and messaging. It also handles foreign servers; their users can ask to join these public rooms. This server interacts with other servers and manages their users requests for registration and event updates in public rooms via the server-to-server communication API. Our OCaml implementation follows the Matrix specification standard. From this, we extract the parts describing the subset of Matrix components that we choose to implement for our OCaml Matrix MVP (Minimum Viable Product). However, the MVP applies its constraints while taking into account that other servers would not be aware of them by using errors/rights restrictions provided by the Matrix standard.

We also implemented an OCaml-CI client that communicates with the Matrix servers via the client-server API. This client implements a subset of the actions defined in the specification and is meant to be used as a bot only (and would therefore not need to drift apart from this subset). The OCaml-CI client was specifically designed to allow an easy implementation for our OCaml server, but it is totally compatible with other Matrix homeservers. We tested the integration of the OCaml-CI client with both Synapse and our ocaml-matrix server, and we used it for testing throughout the ocaml-matrix server implementation.

For now, we’ve only given the OCaml Matrix access to public rooms because they don’t require the end-to-end encryption protocol. Nevertheless, we define support for encrypted communication via the Key module, and we note that most of the encryption algorithms used by the end-to-end encryption protocol are available in MirageOS unikernels via the mirage-crypto library.

We deployed the ocaml-matrix server as an end-to-end application and converted it into the unikernel format. The process of unikernel deployment enables theocaml-matrix unikernel’s compatibility to run on various platforms in isolation, increasing the security level of the Matrix server. The unikernel format of the Matrix server is completed for Unix and in the final stages for the platforms ported by Solo5. It is noteworthy to say that throughout the stage of ocaml-matrix unikernel deployment, we’ve had our share of dream-ing. Going through this experience was a game changer.

Matrix Resurrections (Future Work)

Although we’re thrilled about the progress thus far, there is still much work to do. We plan to revive the OCaml Matrix to improve or add certain features. First, we will add user access to private rooms with end-to-end encryption and more authentication methods that follow Matrix specifications and GDPR recommendations. We will also adopt a methodology for testing and benchmarking for both the ocaml-matrix client and server, integrate the ocaml-matrix codebase into OCaml Multicore, create other ocaml-matrix unikernel deployments, and evaluate the security model provided in the Matrix specifications. Finally, we’ll update and complete the implementation according to the latest Matrix specifications.

Conclusions

Having said all of the above, we invite you to decide whether the Matrix name comes from the provided federated structure in which all communication can be matrixed together or from the idea that it's creating a virtual world that is sustained by the users plugged into it. Do you want to know the truth behind the Matrix? It’s up to you. Will you choose the blue pill or the red pill?

Tarides Sponsors 12th Annual Journées Franciliennes

tarides — 2022-06-02T00:00:00-00:00

Tarides is proud to sponsor the 12th annual programming contest Journées Franciliennes de Programmation! On the 31st of May 2022, students from three different Parisian universities met at La Sorbonne University to engage in some friendly but lively competition.

Bachelor students from La Sorbonne (Paris 6), Paris Cité (Paris 7), and Paris Saclay (Paris 11) participated in a day-long programme creating solutions to a variety of problems. The aim of the competition was not that participants needed to demonstrate detailed knowledge on specific areas of programming, but rather that they applied their combined knowledge of programming usefully. At the end of the day, participants were awarded points based on the problems they’d solved during the day and the winners were announced.

The event was organised by teachers and researchers of computer science, some of whom specialise in OCaml. It was a great opportunity for students to experiment with OCaml under the guidance and supervision of experienced programmers.

Tarides provided computer science books for the participants, along with some fun Tarides swag!

+ + + + +

OCaml.org Reboot: User-Centric Design & Content

tarides — 2022-05-02T00:00:00-00:00

Tarides is pleased to announce the launch of the updated community site, ocaml.org.

Over the past year and a half, we have supported and collaborated with members of the OCaml community on the creation of an updated community website. We are proud to present new features and improvements that will benefit both existing and new generations of OCaml users.

Features

Some of the quality-of-life improvements that users can expect from this update include:

Package documentation site which contains the documentation of every version of every OCaml package
Job board to list job opportunities from the community
Syndicated blog that links to blog articles from the community and offers original blog posts
Success stories that explore how notable companies solve real-world challenges using OCaml
New documentation site which aggregates resources and tutorials to learn OCaml
OCaml playground to try OCaml directly in the browser

The Road So Far

We have worked hard to address concrete requirements from users and provide solutions for the new website. The 2020 OCaml Community Survey highlighted several areas to improve, resulting in new features and content being added.

The survey concluded that the original site lacked easily accessible package documentation, and that job applicants and employers had a difficult time connecting. To address this, we decided early on to include both a job board and a fully-incorporated package documentation page. The job board now provides a place where job seekers can discover opportunities in OCaml, and employers can look for applicants. The package documentation page allows users to find, explore, and compare documentation all conveniently located in one place. Additionally, the team wanted to improve site navigation. This included ensuring easy pathfinding between related topics, together with a focus on improving overall accessibility, allowing successful navigation within the site.

Taking the perspective of different users of the site inspired the creation of brand-new content like Success Stories that highlight ways professionals, academics, and others use OCaml to solve hard problems, create impact, and foster collaboration. It also inspired the new area for tutorials and guides, as well as the OCaml playground, both aimed at making learning OCaml easier and more engaging.

Looking Ahead

There is still plenty of room for improvement and new ideas, and now is a great time for the community to be more involved. We’d like everyone in the community to participate in improving the site and we have created a contribution guide to make the process easier. Please reach out on the issue tracker with ideas and suggestions. We are especially looking for people to help maintain and run the website, and improve the content and general user-experience to help grow our community even more!

To learn more about the reboot and how to get involved please read the original Discuss post.

Lightning Fast with Irmin: Tezos Storage is 6x faster with 1000 TPS surpassed

tarides — 2022-04-26T00:00:00-00:00

Over the last year, the Tarides +storage team has been focused on scaling the storage layer of Octez, +the most popular node implementation for the Tezos blockchain. With +the upcoming release of Octez v13, we are reaching our performance goal of +supporting one thousand transactions per second (TPS) in the +storage layer! This is a 6x improvement over Octez 10. Even better, this +release also makes the storage layer orders of magnitude more stable, +with a 12x improvement in the mean latency of operations. At the +same time, we reduced the memory usage by 80%. +Now Octez requires a mere 400 MB of RAM to bootstrap nodes!

In this post, we'll explain how we achieved these milestones thanks to +Irmin 3, the new major release of the MirageOS-compatible +storage layer developed and maintained by Tarides and used by Tezos. +We'll also explain what this means for the Tezos community now and +in the future.

As explained by a recent post on Nomadic Labs +blog, +there are various ways to evaluate the throughput of Tezos. Our +purpose is to optimise the Tezos storage and identify and fix +bottlenecks. Thus, our benchmarking setup replays actual data (the +150k first blocks of the Hangzhou Protocol on Tezos Mainnet, +corresponding to the period Dec 2021 – Jan 2022) and explicitly +excludes the networking I/O operations and protocol computations to +focus on the context I/O operations only. Thanks to this setup we +managed to identify, fix, and verify that we removed the main +I/O bottlenecks present in Octez:

+ + + + +

+
Comparison of the Transactions Per Second (TPS) performance between Octez 10, +11, 12 and 13 while replaying the 150k +first blocks of the Hangzhou Protocol on Tezos Mainnet¹. +Octez 13 reaches 1043 TPS on average which is a 6x improvement over Octez 10.
+

Merkle databases: to index or not to index

A Tezos node keeps track of the blockchain state in a database called the +context. For each block observed by the node, the context stores a +corresponding tree that witnesses the state of the chain at that +block.

Each leaf in the tree contains some data (e.g., the balance of a particular +wallet) which has a unique hash. Together these leaf hashes uniquely determine +the hashes of their parent nodes all the way up to the root hash of the tree. +In the other direction – moving down the tree from the root – these hashes form +addresses that allow each node to later be recovered from disk. In the Octez +node, the context is implemented using Irmin, an open-source OCaml +library that solves exactly this problem: storing trees of data in which each +node is addressed by its hash.

As with any database, a crucial aspect of Irmin's implementation is its +index, the component that maps addresses to data locations +(in this case, mapping hashes to offsets within a large append-only data file). +Indexing each object in the store by hash has some important advantages: for +instance, it ensures that the database is totally +deduplicated and enables fast random access to any +object in the store, regardless of position in the tree.

As discussed in our irmin-pack post, the context index was +optimised for very fast reads at the cost of needing to perform an expensive +maintenance operation at regular intervals. This design was very effective in +the early months of the Tezos chain, but our recent work on benchmarking the +storage layer revealed two problems with it:

+
content-addressing bottlenecks transaction throughput. Using hashes as +object addresses adds overhead to both reads and writes: each read requires +consulting the index, and each write requires adding a new entry to it. At +the current block rate and block size in Tezos Mainnet, these overheads are +not a limiting factor, but this will change as the protocol and shell become +faster. Our overall goal is to support a future network throughput of 1000 +transactions per second, and doing this required rethinking our reliance on +the index.
+
+
maintaining a large index impacts the stability of the node. The larger +the index becomes, the longer it takes to perform regular maintenance +operations on it. For sufficiently large contexts (i.e., on archive nodes), +the store may be unable to perform this maintenance quickly enough, leading +to long pauses as the node waits for service from the storage layer. +In the context of Tezos, this can lead to users occasionally exceeding the +maximum time allowed for baking or endorsing a block, losing out on the +associated rewards.
+

Over the last few months, the storage team at Tarides has been hard at work +addressing these issues by switching to a minimal indexing strategy in the +context. This feature is now ready to ship, and we are delighted to present the +results!

Consistently fast transactions: surpassing the 1000 TPS threshold

The latest release of Irmin ships with a new core feature +that enables object addresses that are not hashes. This feature unlocks many +future optimisations for the Octez context, including things like automatic +inlining and layered storage. Crucially, it has allowed us to switch to using +direct pointers between internal objects in the Octez +context, eliminating the need to index such objects entirely! This has two +immediate benefits:

+
read operations no longer need to search the index, improving the overall +speed of the storage considerably;
+
+
the index can be shrunk by a factor of 360 (from 21G to 59MB in our tests!). We now only need to index +commit objects in order to be able to recover the root tree for a given +block at runtime. This "minimal" indexing strategy results in indices that +fit comfortably in memory and don't need costly maintenance. As of Octez 13, +minimal indexing is now the default node +behaviour².
+

So what is the performance impact of this change? As detailed in our recent +post on replay benchmarking, we were able to isolate and +measure the consequences of this change by "replaying" a previously-recorded +trace of chain activity against the newly-improved storage layer. This process +simulates a node that is bottlenecked purely by the storage layer, allowing us +to assess its limits independently of the other components of the shell.

For these benchmarks, we used a replay trace containing the first 150,000 +blocks of the Hangzhou Protocol deployment on Tezos Mainnet (corresponding to +the period December 2021 – January 2022)³.

One of the most important metrics collected by our benchmarks is overall +throughput, measured in transactions processed per second (TPS). In this +context, a "transaction" is an individual state transition within a particular +block (e.g., a balance transfer or a smart contract activation). We +queried the TzStats API in order to determine the number +of transactions in each block and thus, our measured transaction throughput. +As shown in the graph above, doing this for the last few releases of Octez +reveals that storage TPS has skyrocketted from ~200 in Octez 12 to more than +1000 in Octez 13! 🚀

As a direct consequence, the total time necessary to replay our Hangzhou trace +on the storage layer has decreased from ~1 day to ~4 hours. We're nearly 6 +times faster than before!

+ + + + +

+
Comparison of CPU time elapsed between Octez 10, 11, 12, and 13 while replaying the 150k +first blocks of the Hangzhou Protocol on Tezos Mainnet¹. While Octez 10 took 1 day to complete +the replay, Octez 13 only takes 4 hours and is nearly 6 times faster than +before!
+

Overall throughput is not the only important metric, however. It's also +important that the variance of storage performance is kept to a minimum, to +ensure that unrelated tasks such as endorsement can be completed promptly. To +see the impact of this, we can inspect how the total block time varies +throughout the replay:

+ + + + +

+
Comparison of block time latencies between Octez 10, 11, 12, and 13 while replaying the 150k +first blocks of the Hangzhou Protocol on Tezos Mainnet¹. Octez 13's mean block validation time is +23.2 ± 2.0 milliseconds while Octez v10 was down from 274 ± 183 milliseconds +(and a worst-case peak of 800 milliseconds!). This 12x improvement in +opearation's mean latency leads to much more consistent endorsement rights for bakers.
+

Another performance metric that has a big impact on node maintainers is the +maximum memory usage of the node, since this sets a lower bound on the +hardware that can run Octez. Tezos prides itself on being deployable to very +resource-constrained hardware (such as the Raspberry Pi), so this continues +to be a focus for us. Thanks to the reduced index size, Octez 13 greatly +reduces the memory requirements of the storage layer:

+ + + + +

+
Comparison of maximal memory usage (as reported by getrusage(2)) between +Octez 10, 11, 12, and 13 while replaying the 150k first blocks of the Hangzhou +Protocol on Tezos Mainnet¹. The peak memory usage is x5 less +in the Octez 13 storage layer compared to Octez 10**, owing to the +significantly reduced size of the index. 400 MB of RAM is now enough +to bootstrap Octez 13!
+

Finally, without an index the context store can no longer guarantee to have perfect object deduplication. Our tests and benchmarks show that this choice has relatively little impact on the context size as a whole, particularly since it no longer needs to store an index entry for every object!

+ + + + +

+
Comparison of storage size between Octez 10, 11, 12, and 13 while replaying the 150k +first blocks of the Hangzhou Protocol on Tezos Mainnet¹. Octez 13's uses similar disk resources +than previous versions: the duplicated data is fully compensated by the +reduced indexed size.
+

What this means for users of the Octez shell:

The general I/O performance of the storage layer is vastly improved, as +the storage operations are 6 times faster and a have 12 times lower mean +latency while the memory usage is divided by 5.
In particular, this mode eliminates the risk of losing baking rewards +due to long index merges.

Migrating your Octez node to use the newer storage

Irmin 3 is included with Octez v13-rc1, which has just been released today. +The storage format is fully +backwards-compatible with Octez 12, and no migration process is required to +upgrade.

Newly-written data after the shell upgrade will automatically benefit from the +new, direct internal pointers, and existing data will continue being read as +before. Performing a bootstrap (or importing a snapshot) with Octez 13 will +build a context containing only direct pointers. Node operators should upgrade +as soon as possible to benefit.

The future of the Octez storage layer

Irmin 3 is just the beginning of what the Tarides storage team has in store for +2022. Our next focus is on implementing the next iteration of the layered +store, a garbage collection strategy for rolling nodes. Once this has landed, +we will collaborate with the Tarides Multicore Applications team to help +migrate Octez to using the newly-merged Multicore OCaml.

If this work sounds interesting, the Irmin team at Tarides is currently +hiring!

Thanks for reading, and stay tuned for future updates from +the Irmin team!

Our benchmarks compare Octez 10.2, 11.1, 12.0, and 13.0-rc1 by replaying the 150k first blocks of the Hangzhou Protocol on Tezos Mainnet (corresponding to the period Dec 2021 – Jan 2022) on an Intel Xeon E-2278G processor constrained to use at most 8 GB RAM. Our benchmarking setup explicitly excludes the networking I/O operations and protocol computations to focus on the context I/O operations only. Octez 10.2 uses Irmin 2.7.2, while both Octez 11.1 and 12.0 use Irmin 2.9.1 (which explains why the graphs are similar). Octez v13-rc1 uses Irmin 3.2.1, which we just released this month (Apr 2022).↩
The trade-off here is that without an index the context store can no longer guarantee to have perfect deduplication, but our testing and benchmarks indicate that this has relatively little impact on the size of the context as a whole (particularly after accounting for no longer needing to store an index entry for every object!).↩
To reproduce these benchmarks, you can download the replay trace we used here (14G). This trace can be replayed against a fork of lib_context available here.↩

Tarides Partners with 50inTech!

tarides — 2022-04-19T00:00:00-00:00

50inTech

Tarides is proud to have been recognised by 50inTech and featured on their website! 50inTech’s mission is to achieve a 50% representation of women in tech by 2050.

To this end, 50inTech runs several amazing initiatives that generate opportunities for women looking to have successful careers in tech. Their job board matches talented women with inclusive companies that are hiring, the 50inTech Gender Score helps European companies measure their level of gender-inclusion, and their free virtual bootcamps provide their network of 15,000 women in Europe with crucial networking and mentoring opportunities.

Partnership

Tarides has been selected as an “inclusive company,” based on metrics including work-life balance, equal pay, fair career path, and diversity and inclusion policies. We are incredibly proud to be recognised in this way, and we will continue to invest in programs and initiatives to further increase diversity and inclusion.

Our partnership with 50inTech connects us with a highly-skilled, diverse set of people, and we hope that this collaboration will help us achieve our target of filling 50% of Tarides’s tech roles with women. Currently that number is 20%, and we’d like to increase it!

Read more about our diversity and inclusion goals, and see our open positions on 50inTech’s website.

Previous Efforts

As Tarides has grown, we have made continuous efforts towards making OCaml and our place within its community more inclusive and diverse. As our CEO Gemma Gordon says, “Different opinions, experiences, backgrounds, and strategies are essential for innovation and vital to the success of Tarides.”

In this vein, we have supported initiatives such as: Outreachy, where we sponsor three paid remote internships per quarter for people experiencing systemic bias and underrepresentation in the tech industry; Ada Tech School, which is a programming school that facilitates greater access to programming positions and promotes the feminisation of tech; and SheCanCode, whose mission is to close the tech gender gap. All of these enterprises help women enter, remain, and excel in the tech industry.

Tarides remains committed to inclusivity and continuously looks for ways to reach out to new groups, gain new perspectives, and diversify our workforce. Read more about our mission to support women in tech on 50inTech’s website.

What's New in MirageOS 4!

tarides — 2022-04-14T00:00:00-00:00

MirageOS 4.0 Release Week

Tarides is thrilled to see the great responses to MirageOS +4.0 and the excitement +that’s building across the community. We’re proud to have played an +important part in its development and release, bringing great tools +and opportunities to OCaml developers. If you haven’t kept up with +what’s been going on since the release, here is a summary of several +articles posted by various OCaml users.

Cross-Compilation

The MirageOS 4.0 update brings with it a major change in its build +system to support the Dune build system. +Tarides has been working on this feature since 2019, +iterating on various design solutions in the mirage tool with +mirage/mirage/#, +mirage/mirage#979, +mirage/mirage/#1020, +mirage/mirage#1024, +mirage/mirage#1153, and +finally miarge/mirage#1226. +This incremental process resulted in making several contributions to +upstream OCaml for features and tools required to support +the flexible building of MirageOS libraries: for +instance, adding support for virtual library and +variants in Dune +with ocaml/dune#1900, +ocaml/dune#2098, and +ocaml/dune#2169; or the +development or a new opam plugin to manage +mono-repositories. We +are happy to see it released to all with Mirage 4.0.

What makes Dune a great option to build MirageOS is that it allows for +customisable cross-compilation flags to compile MirageOS to different +architectures. Using Dune also enables developers to use the Merlin +tool to access a rich set of IDE features when writing +applications. It unlocks a new development workflow based on +opam-monorepo, which downloads all the unikernel dependencies into a +single Dune workspace. Having a single workspace containing all of the +unikernel’s code lets developers edit code anywhere in the stack, +which makes work like debugging libraries and improving APIs a faster +and more enjoyable experience. In his excellent article on build +contexts in MirageOS +4.0, Lucas +Pluvinage goes into detail about how to use the new cross-compilation +features to build MirageOS unikernels for new architectures.

Email in OCaml & Mr. MIME

Mr. MIME is an OCaml library that aims to give its users peace of mind +when it comes to the security of their email communications. Mr. MIME +is built on unikernels and deploys them to handle email traffic. At +Tarides, we got a grant from NGI DAPSI to +work on this project, and several of our engineers have been busy +working hard to make it happen.

Several other libraries support the Mr. MIME library and enable it to +transform an email into an OCaml value, then create an email from it +again. An amazing thing about Mr. MIME is its reliability. Using the +hamlet tool, which proposes a +large corpus of emails for Mr. MIME to parse and re-encode, the team +can prove that Mr. MIME doesn’t alter anything in the message between +the parser and the encoder.

The team behind Mr. MIME has also created the library +Colombe that implements the +foundations of an SMTP protocol with the ability to upgrade its flow +to TLS, giving its users an extra layer of security. A goal for the +future is to provide a full SMTP stack that’s able to send and receive +emails.

Mr. MIME also allows its users to manipulate emails through the use of +CLI tools, including +ocaml-dkim, a tool to verify +and sign an email, and +spamtacus, a tool which +analyses the incoming email to determine if it’s spam or not. The +ptt repo contains several more as well.

If you want to find out more information about Mr. MIME, including +details about its architecture, please read Romain Calascibetta’s +article.

MirageOS in Production

The use of MirageOS benefits not only Tarides, but it also enables +several other companies to make their products better. Below are a +couple of examples from Docker and +Robur on how they use MirageOS to their +advantage.

VPN Kit

Docker Desktop is a tool that enables its users to build and share +containerised or isolated applications in either a Mac or Windows +environment. Its main challenge is that running Docker on macOS or +Windows is difficult in terms of compatibility, as Linux primitives +are unavailable on those platforms.

This is where VPN Kit comes in; it uses MirageOS to bridge the gap +between Linux primitives and macOS or Windows by reading the raw +ethernet frames coming out of the Linux VM and translating them into +macOS or Windows high-level syscalls. In this way, MirageOS networking +libraries transparently handle the traffic of millions of containers +every day.

To find out more go read the article “How MirageOS Powers Docker +Desktop” on mirage.io +or +on docker.com.

Robur Projects

Robur uses MirageOS for several of their projects, including OpenVPN, +DNS Projects, and CalDAV. All of these projects are written in OCaml +and are deployed as MirageOS unikernels.

The DNS Projects include the ‘Let’s Encrypt’-Certified DNS solver, a +DNS resolver, and an authoritative DNS server. Robur’s DNS server +ensures that the internet user gets to the right IP address, whilst +its DNS resolver finds the exact server to handle the user’s +request. Only strictly necessary elements are included in order to +keep the codebase as small as possible for security and +simplicity.

CalDAV is the most recent unikernel released by Robur. As the name +implies, CalDAV is a protocol used to synchronise calendars. +Its minimal codebase comes with significant security benefits.

To find out more go read the article “MirageOS Unikernels at Robur” on +mirage.io.

To learn more about MirageOS, take a look at some recent articles at +mirage.io. +If you’re interested in working with Tarides or +incorporating MirageOS tools in your project, please contact us via +our website.

MirageOS 4 Released!

tarides — 2022-03-29T00:00:00-00:00

Tarides is delighted to announce that MirageOS 4 is finally released! As core contributors to the project, we are proud to have been part of the journey to 4.0.

What is MirageOS? +MirageOS is a library operating system that constructs unikernels for fast and secure network applications that work across a variety of cloud computing and mobile platforms. The goal of MirageOS is to give the individual control of their own data and take back control of their privacy.

It achieves these goals in several ways, from securely deploying static website hosting with Let’s Encrypt certificate provisioning and a secure SMTP stack, to ensuring data privacy with decentralised communication infrastructures like Matrix, OpenVPN Servers, and TLS tunnels, as well as using DNS(SEC) Servers for better authentication.

Over the years since its first release in 2013, the Mirage ecosystem has grown to include hundreds of libraries and service millions of daily users, along with several major commercial users that rely on MirageOS to keep their code secure. Examples of this include Docker Desktop’s VPNkit, the Citrix Hypervisor, as well as Robur, Nitrokey, and Tarides itself!

What’s in the New Release? +The new release focuses on better integration with existing ecosystems. For example, it is now much easier to integrate with existing OCaml libraries, as MirageOS 4 is now using dune to build unikernels.

There has also been a major change in how MirageOS compiles projects with the introduction of a new tool called opam-monorepo that separates package management from building the resulting source code. The Opam plugin can create a lock file for project dependencies, download and extract dependency sources locally, and even set up a Dune workspace, which then enables dune build to build everything simultaneously.

The new release also adds systematic support for cross-compilation to all supported unikernel targets, meaning that libraries that use C stubs can now have those stubs seamlessly cross-compiled to a desired target.

To find out more about the new release please read the official release post on Mirage.io.

Keep an eye on mirage.io's blog over the next two weeks for more posts on the exciting new things that come with MirageOS 4.0, starting with “Introduction to Build Contexts in MirageOS 4.0” tomorrow!

Secure Virtual Messages in a Bottle with SCoP

tarides — 2022-03-08T00:00:00-00:00

People love to receive mail, especially from loved ones. It’s heartwarming to read +each word as their thoughts touch our deepest feelings. Now imagine someone else +reading those private sentiments, like a postal worker. Imagine how violated they’d +feel if their postal carrier handed them an open letter with a knowing smile. Of course, +people trust that postal employees won’t read their personal correspondence; +however, they regularly risk their privacy when sending emails, images, and messages.

Around 300 billion emails traverse the Internet every single day. They travel +through portals with questionable security, and the messages often contain +private or sensitive data. Most online communication services are composed of +multiple components with complex interactions. If anything goes wrong, it +results in critical security incidents. This leaves an unlocked door for +malicious hackers to breach private information for profit or just for fun. +Since it takes considerable technical skills and reliable infrastructure to +operate a secure email service, most Internet users must +rely on third-parties operators. In practice, there +are only a few large companies that can handle communications with the +proper security levels. Unfortunately for regular people, these companies +profit from mining their personal data. Due to this global challenge, Tarides +focused their efforts to address these issues and find solutions to protect +both personal and professional data.

An Innovative Solution

Our work resulted in the project "Secure-by-Design Communications Protocols" +(SCoP), a secure, easily deployable solution to preserve users' privacy. In +essence, SCoP puts your messages in a secure, virtual ‘bottle’ to protect it +from invasive actions. This bottle represents a secure architecture using +type-safe languages and unikernels for both email and instant messaging. +We mould unikernels (specialised applications that +run on a VM) into refined meshes linked by TLS-firm communication pipes, +as depicted in the image below.

+ + + + +

The SCoP virtual bottle creates a trustworthy information flow where dedicated +unikernels ensure security for communication from origin to destination. We +carefully design every component of SCoP as independent libraries, using +modern development techniques to avoid the common reported threats and flaws. +The OCaml-based development enables this safe online +environment, which eliminates many exploited security pitfalls. Moreover, +our SCoP project comes with energy-efficient consumption provided by the +lightweight and low-latency design components.

We mostly focused on the sender’s side, securing the message inside the SCoP +bottle. For instant messages, we created a capsule with a +Matrix client library, +and for emails we based our bottle on the SMTP protocol +and Mr. MIME. For further protection, +we developed the bottle’s ‘cork’ with the +Hamlet email corpus.

The SCoP Processes

First, we generated Hamlet, a collection of emails to test our parser +implementation against existing projects, to ensure that they kept equivalence +between the encoder and decoder. After we successfully parsed and encoded one +million emails, we used Hamlet to stress-test our SMTP stack.

Secondly, we created an SMTP extension mechanism and support for SPF, including +an implementation for DMARC, a security framework in addition to DKIM and SPF. +We completed four components: SPF, DKIM, SMTP, and Mr. MIME, which can generate +a correctly-signed email, signatures, and the DKIM field containing the signatures.

In essence, we designed the SMTP sender bottle with a mesh of unikernels connected +via secured communication pipes. The SMTP Submission Server unikernel receives +the sender’s authentication credentials against the secured database maintained +by Irmin. After it confirms the credentials, it sends the email for sealing +(via a TLS pipe) to the DKIM signer. Then the DKIM signer unikernel, responsible +for handling IP addresses, communicates via the nsupdate protocol with the Primary +DNS Server. The DKIM signer places the sender’s and receiver’s addresses on the email, +seals it with the DKIM signature, and sends it to the SMTP relay for distribution. +The SMTP relay unikernel communicates with the DNS resolver unikernel to locate the +receiver by the DNS name, then it coordinates this location with the Irmin database +to verify the authorization according to the SPF protocol. After all these checks +have passed, the signed and sealed email is secured in the SCoP bottle and launched +through Cyberspace.

Next, we developed the Matrix protocol’s client library, and we used it to enable +notifications from the CI system, testing all the new OCaml packages. We also +designed an initial PoC for a Matrix’s server-side daemon.

We made significant progress in deploying DNSSEC, a set of security extensions over +DNS. While we completed our first investigation into the DNSSEC prototype, we also +discovered several issues, so we addressed those as lessons learned.

Finally, we completed the SCoP bottle with +the email receiver, which Spamtacus (the +Bayesian spam filter) guards against spam intruders. Furthermore, the +OCaml-Matrix server represents +our solution to take care of the instant communication in the Matrix federation.

A Secure-by-Design SMTP Stack

We researched state-of-the-art email spam filtering methods and identified machine +learning as the main trend. We followed this path and equipped our email architecture +with a spam-filter unikernel, which uses a Bayesian method for supervised learning +of spam and acts as a proxy for internet communication in the SMTP receiver. This +spam filter works in two states: preparation, where the unikernel detects spam, +and operation, where the unikernel integrates into the SMTP receiver unikernel +architecture to filter spam emails. Our spam-filter unikernel can also be used +independently as an individual anti-spam tool to help enforce GDPR rules and protect +the user’s privacy by preventing spam-induced attacks, such as phishing.

We integrated our spam filter into a unikernel positioned at the beginning +of the SMTP receiver stack. This acts as a first line of defence in an eventual +attack targeting the receiver in order to maintain functionality. The spam-filter +unikernel can be extended to act as an antivirus by analysing the email attachment +for certain features known to characterise malware. We’ve already set the premises +for the antivirus by using a prototype analysis of the email attachments. Moreover, +the spam-filter unikernel can contribute with a list of frequent spammers to the +firewall, which we plan to add into the SMTP receiver as the next step in our development of SCoP.

How the Technology Works

DKIM, SPF, and DMARC are three communication protocols meant to ensure email +security by verification of sender identity. The latest RFC standards for +DKIM, SPF, and DMARC are RFC8463, RFC7208, and RFC7489, respectively.

DKIM provides a signer protocol and the associated verifier protocol. DKIM +signer allows the sender to communicate which email it considers legitimate. +Our implementation of the DKIM verifier is associated with the SMTP receiver, +it follows the RFC8463 standard and supports the ED25519 signing algorithm, +i.e., the elliptic curve cryptography generated from the formally verified +specification in the fiat project from MIT.

SPF is an open standard that specifies a method to identify legitimate mail +sources, using DNS records, so the email recipients can consult a list of IP +addresses to verify that emails they receive are from an authorised domain. +Hence, SPF is functioning based on the blacklisting principle in order to +control and prevent sender fraud. Our implementation of the SPF verifier +follows the RFC7208 standard.

DMARC (Domain-based Message Authentication, Reporting, and Conformance) enables +a sender to indicate that their messages comply with SPF and DKIM, and applies +clear instructions for the recipient to follow if an email does not pass SPF or +DKIM authentications (reject, junk, etc.). As such, DMARC is used to create +domain reputation lists, which can help determine the actual email source +and mitigate spoofing attacks. Our implementation of the DMARC verifier is +integrated in the SMTP receiver and follows the RFC7489 standard.

Our secure-by-design SMTP stack contains the DKIM/SPF/DMARC verifier unikernel +on the receiver side. This unikernel verifies the email sender’s DNS +characteristics via a TLS communication pipe, and in case the DNS verification +passes, the spam-labelled email goes to the SMTP relay to be dispatched to the +email client. However, in case the DNS verification doesn’t pass, we can use +the result to construct a DNS reputation list to improve the SMTP security +via a blacklisting firewall.

Matrix Server

The Matrix server in our OCaml Matrix implementation manages clients who are +registered to rooms that contain events. These represent client actions, such +as sending a message. Our implementation follows the Matrix specification +standard. From here, we extracted the parts describing the subset of the Matrix +components we chose to implement for our OCaml Matrix server MVP. The OCaml +implementation environment provides secure-by-design properties and avoids +various vulnerabilities, like the buffer overflow recently discovered that +produces considerable information disclosure in other Matrix implementations, +e.g., Element.

The Matrix clients are user applications that connect to a Matrix server via the +client-server API. We implemented an OCaml-CI client, which communicates with the +Matrix servers via the client-server API and tested the integration of the OCaml-CI +communication with both Synapse and our OCaml Matrix server. Please note that our +OCaml Matrix server supports a client authentication mechanism based on user name +identification and password, according to the Matrix specification for authentication +mechanisms.

Spam Filter

We researched the state of the art in email spam filtering and we identified machine +learning as the main trend. We follow this trend and we equip our email architecture +with a spam filter unikernel, which uses a Bayesian method for supervised learning of +spam and acts as a proxy to the internet communication in the SMTP receiver. The spam +filter implementation works in two stages: preparation, when the unikernel is trained +to detect spam, and operation, when the unikernel is integrated into the SMTP receiver +architecture of unikernels to filter the spam emails. It is worth mentioning that the +spam filter unikernel can be used independently as an individual anti-spam tool to help +enforce the GDPR rules and protect the user's privacy by preventing spam induced attacks +such as phishing.

We integrate the spam filter into an unikernel positioned at the beginning of the +SMTP receiver stack as the first line of defence in an eventual attack targeting the +receiver. In this situation, the unikernel format provides isolation of the attack and +allows the SMTP receiver to maintain functionality. The spam filter unikernel can be +extended to act as an antivirus by analysing the email attachment for certain features +that are known to characterise malware. We have already set the premises for the antivirus +by a prototype analysis of the email attachments. Moreover, the spam filter unikernel could +contribute with a list of frequent spammers to the firewall, which is planned to be added +into the SMTP receiver, as the next step in future work.

The DAPSI Initiative

Much of the SCoP project was possible thanks to the DAPSI initiative. +They gave Tarides the incentive to further explore an open and secure infrastructure for +communication protocols, especially emails. First, DAPSI supported our team by providing +necessary financing, but their contribution to our project’s prosperity runs much deeper +than funding. DAPSI facilitated multiple coaching sessions that helped broaden our horizons +and established reachable goals. Notably, their business coaching enabled us to identify +solutions for our market strategy. Their technical coaching and training offered access +to data portability experts and GDPR regulations, which opened our perspective to novel +trends and procedures. Additionally, DAPSI helped raise our visibility by organising public +communications, and DAPSI’s feedback revealed insights on how to better exploit our project’s +potential and what corners of the cyber-ecosystem to prioritise. We are deeply grateful to +DAPSI for their support and backing, and we’re thrilled to have passed Phase 2!

Up Next for SCoP

We’re excited to further develop this project. We’ll be experimenting with deploying +unikernels on a smaller chipset, such as IoT. We’d also like to research secure data +porting in other domains such as journalism, law, or banking.

Of course we’ll be maintaining each of the SCoP components in order to follow the latest +available standards and state-of-the-art technology, including periodical security +analyses of our code-base and mitigation for newly discovered vulnerabilities.

As in all of our work at Tarides, we strive to benefit the entire OCaml community and beyond. +Please find more information on SCoP through our blog posts: +DAPSI Initiative +and DAPSI Phase 1.

+ + + + +

Segfault Systems Joins Tarides

tarides — 2022-03-01T00:00:00-00:00

We are delighted to announce that Segfault Systems, a spinout from IIT-Madras, +is joining Tarides. Tarides has worked closely with Segfault Systems over the +last couple of years, most notably on the award-winning Multicore OCaml project +and the upstreaming plans for OCaml 5.0. This alliance furthers the goals of +Tarides, bringing the compiler and benchmarking expertise of the Segfault team +directly into the Tarides organisation.

KC Sivaramakrishnan, CEO & CTO of Segfault Systems says that “Segfault Systems +was founded to secure the foundations of scalable systems programming in OCaml. +We have successfully incorporated cutting-edge research on +concurrent and +parallel programming into OCaml. This +addresses the long-standing need of OCaml developers to utilise the widely +available multicore processing on modern machines. Tarides is at the forefront +of OCaml developer tooling and platform support, and we are excited to join the +team to make OCaml the best tool for industrial-strength concurrent and parallel +programming.”

“We’re thrilled to have the Segfault Systems team join Tarides,” says Thomas +Gazagnaire, CTO of Tarides. “They have been integral to the success of the +Multicore OCaml project, which has combined cutting edge research and +engineering with consistent communication, promoting Multicore OCaml as an +upstream candidate to the core developer team, as well as +publishing monthly reports +for the wider community. We look forward to working with our new partners to +make OCaml the tool of choice for developers.”

All of Segfault Systems’ existing responsibilities and open-source commitments +will migrate over to Tarides, where work will continue towards the three main +objectives in 2022:

Releasing OCaml 5.0 with support for domains and effect handlers
Supporting the ecosystem to migrate the OCaml community over to OCaml 5.0
Improving developer productivity for OCaml 5.0 by releasing the best platform +tools

OCaml 5.0

The next major release of OCaml, version 5.0, will feature primitive support for +parallel and concurrent programming through domains and effect handlers. The +goal is to ensure that the fine balance that OCaml has struck between ease of +use, correctness and performance over the past 25 years continues into the +future with these additional features.

Domains enable shared-memory parallel programming allowing OCaml programs to run +on multiple cores: with domains, OCaml programs will scale better by exploiting +multicore processing. Effect handlers are a mechanism for concurrent +programming: with the introduction of effect handlers, simple direct-style OCaml +code will be flexible, easy to develop, debug and maintain (no more monads for +concurrency!). These features will benefit the entire ecosystem and community, +and we expect it to attract many new users to the language.

As part of the Multicore OCaml project, the team developed +Sandmark, a suite of sequential and +parallel benchmarks together with the infrastructure necessary to carefully run +the programs and analyse the results. Sandmark has been instrumental in +assessing and tuning the scalability of parallel OCaml programs and ensuring +that OCaml 5.0 does not introduce performance regressions for existing +sequential programs compared to OCaml 4.

+ + + + +

Scalability of compute intensive OCaml programs

Sandmark is now run as a nightly service +monitoring the performance of OCaml 5 as it is being developed. Development will +continue to make it even easier to use and more practical by fully integrating +it with current-bench (the continuous +benchmarking system based on OCurrent). +Get in touch if you want to know more.

Ecosystem

At Tarides we want all OCaml users to benefit from the new features that OCaml +5.0 will bring, and this means ensuring that the ecosystem is fully prepared. We +aim to develop and maintain a robust set of libraries that work with domains and +effects, together with a diverse parallel benchmarking and performance profiling +suite to use with OCaml 5 applications. The +first version of Eio, +the effects-based direct-style IO stack for OCaml 5.0, has been released, +generating lots of interesting discussion within the community. Eio not only +makes it easier to develop, debug and maintain applications utilising +asynchronous IO, but is also able to take advantage of multiple cores when +available.

+ + + + +

HTTP server performance using 24 cores

+ + + + +

HTTP server scaling maintaining a constant load of 1.5 million requests per second

The early results are quite promising. An HTTP server based on Eio is able to +serve 1M+ requests/sec on 24 cores, outperforming Go's nethttp and closely +matching Rust's hyper performance. Eio is still heavily under development. +Expect even better numbers for its stable release planned later this year.

The next step is to iterate on the design in collaboration with the community +and our partners. Get in touch if you have +performance-sensitive applications that you'd like to port to Eio, so we can +discuss how the design can meet your needs.

OCaml Platform

In collaboration with community members and commercial funders, Tarides has been +developing and defining the +OCaml platform tool suite for the last +four years. The goal of the platform is to provide OCaml developers with easy +access to high-quality, practical development tools to build any and every +project. We will continue to develop and maintain these tools, and make them +available for OCaml 5. Reach out to us if you +have specific feature requests to make your developer teams more efficient.

This alliance brings the headcount of Tarides up to 60+ people, all working +towards making OCaml the best language for any and every project. +Join us!

OCaml Labs Joins Tarides

tarides — 2022-01-27T00:00:00-00:00

Today I am incredibly delighted to announce that OCaml +Labs, a spinout from the +University of Cambridge, is joining +Tarides. After successfully collaborating on +many OCaml projects over the last four years, this alliance will +combine the expertise of both groups and enable us to bring OCaml, one +of the most advanced programming languages in the world, into +mainstream use. Combining forces will accelerate OCaml development and +its broader adoption. Furthermore, it will bring the security, +portability, and performance of OCaml to a large spectrum of +use-cases: from academic endeavours, such as formal methods and +existing threats within cyber-security, to real-world applications for +climate change, sustainable agriculture, and even space exploration!

All of OCaml Labs’ existing responsibilities and open-source +commitments will migrate over to Tarides, and thanks to how closely +the teams already work, business will continue without interruption to +continuity or delivery. Gemma Gordon will step up as CEO of Tarides, +and I will continue to lead the technological vision and strategy as +CTO. As Prof. Anil Madhavapeddy, founder of OCaml Labs and scientific +advisor of Tarides, points out, “The cutting edge research we +conducted at the University over the past decade has now migrated into +mainline OCaml, and so the ongoing curation and development will now +happen on a commercially supported basis. I’m excited to continue +collaborating on research with Tarides from the University of +Cambridge.” Tarides will continue the work started at OCaml Labs and +invest in the growth, health, and development of OCaml alongside its +wider use-cases.

I am honoured to have the incredible OCaml Labs team - the team who +carefully designed and crafted Multicore OCaml - join Tarides. We +share a similar view that a common plague affects the ever-growing +software industry, namely the bad quality of software and the +omnipresence of bugs. However, this is not a fatal flaw. Tools +developed by OCaml Labs over the years do not compromise on quality, +and they allow dev teams to automatically fix at least 70% of +security +bugs +and 0-days security +exploits. Consequently, +OCaml is a simple yet powerful language that can respond to the many +challenges developers face today. Since Tarides’s inception, we have +envisioned a future where all OCaml applications are easily deployable +as specialised, secure, and energy-efficient +MirageOS unikernels. This alliance is a step +further in that direction. Since OCaml is the language used to develop +MirageOS, Tarides has continuously developed and maintained parts of +the OCaml ecosystem since its creation. Our alliance with OCaml Labs +makes this more evident. The MirageOS ecosystem critically depends on +OCaml, and the OCaml ecosystem benefits from innovations coming from +the MirageOS project. Tarides is therefore fully committed to making +the synergy between OCaml and MirageOS a success.

Several exciting projects related to Multicore OCaml are coming to a +head this year. The OCaml 5.0 release will support multicore and +effects handlers, influencing every aspect of the language and its +ecosystem. The update will significantly improve both performance and +user experience whilst maintaining existing features that make OCaml +the language of choice for building, for instance, verification +software tools. Using the teams’ combined experience and zest for +innovation, Tarides is looking to the future of the OCaml language and +community with excitement. We will continue to push the boundaries of +exploration whilst focusing on what's good for the +community. Therefore, this alliance will complement the commercial +offering of Tarides and contribute to Tarides' mission: empowering +developers, communities and organisations to adopt OCaml as their +primary programming experience by providing training, expertise, and +development services around the OCaml language.

“We are thrilled to be part of an organisation innovating in many +areas around operating systems, distributed systems, and security with +the Irmin distributed store and the MirageOS +unikernel projects,” says Gemma Gordon, CEO of Tarides. “I am +incredibly proud of the people OCaml Labs has collaborated with. We +have been able to build a sustainable open-source community, with +people from various backgrounds all collaborating together. It used to +be that people would have to volunteer their time on OCaml, or work in +academic research. We have created an additional funded path, one that +has increased the diversity and innovation of our community. I’m +excited to continue to be part of a group that brings the best minds +together to solve the many problems the software industry faces +today. I look forward to building a flourishing and sustainable +commercial business with existing Tarides partners as well as +developing new collaborative opportunities.”

This alliance brings the headcount of Tarides up to 60+ people, all +working towards making OCaml the best language for any and every +project. Join our team: https://tarides.com/company

OCaml Labs

OCaml Labs has been at the forefront of innovation in OCaml for +nearly a decade. It was founded at the University of Cambridge by +Prof. Anil Madhavapeddy in 2012 and developed into a spin-out +consultancy company in 2016. OCaml Labs' mission was to push OCaml and +functional programming forward as a platform, making it a more +effective tool for all users (including large-scale industrial +deployments) while at the same time growing the appeal of the language +to broaden its applicability and popularity.

OCaml Labs has been instrumental in developing and maintaining the +OCaml platform for OCaml usage at an industrial scale. OCaml Labs +contributed to the development and maintenance of the +opam package management +ecosystem and of the OCaml community website, https://ocaml.org/, +first launched in 2012. These sites act as hubs for the OCaml community +to showcase the state-of-the-art and facilitate innovation. A new and +improved version of the site has been released under +beta this month. In addition, OCaml Labs' most +significant (and technically complex) project, OCaml Multicore, will +finally come to fruition this year. Work on this project began in +2014, followed by award-winning papers and presentations in 2020 and +the announcement in late 2021 that Multicore will become part of the +mainline OCaml compiler.

Tarides

Tarides is a tech start-up founded in Paris in 2018 by pioneers of +programming languages and cloud computing. They develop a software +infrastructure platform to deploy secure, distributed applications +with strict resource constraints and low-latency performance +requirements. This platform builds upon innovative and open-source +projects such as MirageOS and Irmin and underpins mission-critical +deployments such as +Tezos, +Citrix XenServer, or Docker for +Desktop. In +addition, Tarides uses unikernel technologies and applies the research +done in programming languages to real-world systems to build safe and +performant applications.

Tarides has been part of the Founder program of Station F in 2018. In +addition, it got selected in France for the Concours d’Innovation +i-Lab, organised by +the French Ministry of Higher Education, Research and Innovation in +partnership with Bpifrance. This national contest awards company +creation and innovative technologies. Tarides got awarded during the +FIC +2020, +the leading European cybersecurity event.

'Signals and Threads' Podcast: What is an Operating System?

tarides — 2021-11-23T00:00:00-00:00

November has become MirageOS month! Between the upcoming official MirageOS 4.0 release, making custom Christmas Tree garlands +with MirageOS on a Raspberry Pi, +and now this "What is an Operating System?" podcast (featuring Tarides advisor and core MirageOS maintainer Anil Madhavapeddy), it truly is MirageOS month!

MirageOS can do much more than program a Raspberry Pi for Christmas decor. From +agricultural monitoring to +smart buildings, its +applications cover a wide range of needs. For example, it can also be used in critical pieces of infrastructure where security is of paramount importance, such as the Tezos blockchain. +Combined with Irmin, it allows developers to build secure-by-design, offline-first systems and invert +the current cloud-centric model for designing applications to securely connect physical spaces with extremely low latency +and high bandwidth, using local-area computation capabilities.

You can read the entire transcript here and find links to multiple places to listen through podcast apps.

Tarides & Hyper: Partners in Agricultural Innovation

tarides — 2021-11-18T00:00:00-00:00

We are thrilled to announce a partnership between Tarides and Hyper, a technology provider in the agritech space who’s building +an "operating system for high-performing farms." Indoor and vertical farms are becoming tech businesses that require scalable, +flexible, and easy-to-use tools to facilitate data analysis and thereby increase productivity. According to the State of Indoor +Farming 2020 Report, “40% of vertical and indoor farms are +implementing data analytics and control automation to increase yield and lower cost of production.”

Hyper’s product offers a developer-friendly platform for modern farms to integrate analytics and automation without +worrying about hardware, scaling, or maintenance. There is a natural synergy between Hyper's product and Tarides’s mission +to bring robust and scalable functional systems to the industry. Both teams will be working on technology to help solve +some big problems the world faces.

First, some context about how agriculture affects our environment:
+The world’s population is growing, but the size of our planet is not.

Agriculture is currently responsible for:

75% of the world’s deforestation¹
50% of the world’s habitable land
70% of the world's freshwater withdrawals
26% of the greenhouse gas emissions²

It’s an important time for indoor farming, as it’s already producing twenty times (20x) more yield per area while using 95% less water and +zero (0) pesticides. Plus, indoor farming will reduce the carbon footprint by cutting their food miles in half (50%)³. Vertical farming +complements traditional farming by growing fresh produce for cities in a sustainable way.

Hyper's mission is to simplify the integration of sensors and controllers for data collection and automation. Their roadmap is +focused on implementing real-time computation of metrics for environmental data gathered from farms, prototyping computer vision models +for crop analysis, and automated crop traceability infrastructure. With their data platform, growers can optimise yield and reduce operating +costs by getting access to crop growth metrics and climate automation profiles without a dedicated engineering team.

Hyper's founders are experienced engineers and OCaml hackers who have worked on IoT and data analytics products in the past in the retail, +biotech, and multimedia industries. Utilizing our MirageOS ecosystem, Hyper's sensor networks and cameras are continuously collecting +millions of data points across large-scale farming operations to ensure consistent crop quality, detect issues early, and automate climate +control. Since the technology is offline-first, it enables Hyper to collect this data securely, without the risk of breach or loss, as is +sometimes the case in cloud computing.

As a technological partner, Tarides will help Hyper build a scalable IoT platform that leverages the OCaml ecosystem. Our support will +help Hyper bring the product to the market faster while also contributing to the open-source IoT ecosystem for MirageOS and related +projects. Hyper’s IoT platform will be fully open-source next year and will focus on implementing better support for MQTT, CoAP, and other protocols.

A new version of MirageOS will be released in November, so this exciting announcement couldn't come at a better time! Hyper’s use of +the MirageOS ecosystem to build its data intelligence product is yet another real-world example that displays the +power and efficiency of MirageOS.

It’s truly exciting technology because we’re at a point in history where even the most adamant climate change critics have begun to admit +the climate is indeed changing. There are things we can each do to contribute, so Tarides is proud to collaborate with a company that’s +actively seeking solutions to help reduce the amount of greenhouse gas emissions while increasing productivity.

Hyper currently has production deployments in vertical and indoor farms in the UK and East Africa and is planning on scaling the +operations in the coming months.

SOURCES

MirageOS Workshop: Working with the Raspberry Pi 4

tarides — 2021-11-11T00:00:00-00:00

Earlier this week, Romain Calascibetta hosted an in-house MirageOS workshop for employees, both locally and remotely +around the world. This interactive workshop taught participants how to build an operating system on a Raspberry +Pi 4 using MirageOS. They got to create their own OS and play with projects, like one they dubbed GuirlandeOS for +which they programmed an LED garland to trim their Christmas tree, creating their own customized light show! There will +be a dedicated blog to GuirlandeOS soon.

That’s one fun example of what can be done with MirageOS and Raspberry Pi, but the possibilities are endless. +For example, one could use this dynamic pair to create solar-powered websites (something we’ll hear more about next week).

The spirit of MirageOS is that anyone can integrate it, even if they don't work at Tarides. Although the +workshop was only for employees, MirageOS is for everyone! Since it's autonomous from Tarides, we encourage you +to play with MirageOS and see what you can create.

Romain has opened the contents of his informative workshop to the world! Follow along with +his slides, +which will walk you through the MirageOS toolchain to create your very own projects. You can read more about the +Raspberry Pi process in this repo.

Early next week, we’ll continue with MirageOS month by showcasing a podcast where Anil Madhavapeddy +talks about those solar-powered websites and more!

Happy hacking!

MirageOS 4.0 Preview Live Presentation

tarides — 2021-11-09T00:00:00-00:00

The official release of MirageOS 4.0 quickly approaches! Learn about some general MirageOS concepts and get a +sneak park at the forthcoming changes in MirageOS 4.0 during a LIVE presentation today at 15h CET.

Lucas Pluvinage will lead you through a live-streaming presentation to acquaint you with MirageOS 4.0. You’ll learn +what kinds of problems MirageOS can solve and about Functoria, the compilation model. Then Lucas will also discuss the switch +to the Dune build system and how that enables cross-compilation, not to mention the creation of new compilation targets, +such as the Raspberry Pi!

Watch the live presentation today at 15h CET. Just go to: https://meet.google.com/iqy-urht-rcn, or dial ‪(FR) +33 1 87 40 43 45‬ with +the PIN: ‪288 878 885‬#. You can also find other phone numbers here: https://tel.meet/iqy-urht-rcn?pin=3736259978366 if you're not in France.

This presentation will be a great introduction to Romain Calascibetta's forthcoming MirageOS workshop tomorrow. +Check back on this blog and follow us on Twitter to find out more about how to watch +and participate in this informative workshop tomorrow!

Until then, check out Lucas's presentation today at 15h CET.

SCoP Passed Phase 1 of the DAPSI Initiative!

tarides — 2021-10-14T00:00:00-00:00

In April, we announced that the DAPSI initiative accepted +the proposal for our Secure-by-Design Communication Protocols (SCoP) project. Today, we are thrilled to announce that SCoP has passed the initiative’s Phase 1, +and we are now on our way to Phase 2!

SCoP is an open, secure, and resource-efficient infrastructure to engineer a modern basis for open messaging (for existing and emerging protocols) +using type-safe languages and unikernels—to ensure your private information remains secure. After all, you wouldn’t like your postal carrier reading +your snail mail, so why should emails be any different?

Challenges

To operate an email service requires many technical skills and reliable infrastructure. As a result, only a few large companies can handle emails with +the proper security levels. Unfortunately, the core business model of these companies is to mine your personal data.

The number of emails exchanged every day is expected to reach 333 billion in 2022. That’s a considerable amount of data, much of it private or sensitive, +sent across Cyberspace through portals with questionable security. The ‘memory unsafe’ languages used in most communication services leave far too much room +for mistakes that have serious ramifications, like security flaws that turn into security breaches, leaving your personal or business information vulnerable to +malicious hackers.

Due to this global challenge, we set out to build a simple, secure, easily deployable solution to preserve users' privacy, and we’re making great strides toward +accomplishing that goal. We base our systems on scientific foundations to last for decades and drive positive change for the world. Our robust understanding of +both theory and practice enables us to solve these security problems, so we explore ideas where research and engineering meet at the intersection of the domains +of operating systems, distributed systems, and programming languages.

Every component of SCoP is carefully designed as independent libraries, using modern development techniques to avoid the common reported threats and flaws. +For instance, the implementation of protocol parsers and serializers are written in a type-safe language and tested using fuzzing. Combining these techniques +will increase users' trust to migrate their personal data to these new, more secure services.

Architecture

The architecture of the SCoP communication service is composed of an Email Service based on a secure extension of the SMTP protocol, and a decentralised +real-time communication system based on Matrix.

The SMTP and Matrix protocols implemented in SCoP follow the separation of +concerns design principle, meaning that the SMTP Sender and SMTP Receiver are designed as two distinct units. They’re implemented as isolated micro-services +which run as unikernels. The SMTP Sender, Receiver, and Matrix are all configurable, and each configuration comes with a security risk analysis report to +understand possible privacy risks

Progress

Not only are we on our way to Phase 2 in the DAPSI Initiative, but we’re also proud to report that we’re on track with our +planned milestones!

Our first milestone was to generate a corpus of emails to test our parser implementation against existing projects in order to detect differences +between the descriptions specified in the RFCs. We now have 1 million emails that have been parsed/encoded without any issues! Our email corpus keeps +isomorphism between the encoder and decoder, and you can find it in this GitHub Repo, as we encourage implementors of other languages to use it to improve +their trust in their own implementation.

We set out to implement an SMTP extension mechanism and support for SPF as well as implement DMARC, a security framework, on top of DKIM and SPF for our +second milestone, and we are right on target. To date, we’ve completed four components:

SPF
DKIM
SMTP can send and verify emails
MrMIME can generate the email, then SMTP sends the email (signed by a DKIM private key). We can correctly sign an email, generate a signature, and the DKIM field containing the signature. When the email is received, we check the DKIM signature and the SPF metadata.

For our third milestone, we set out to implement DNSSEC, a set of security extensions over DNS. This security layer verifies the identity of an email sender +through DKIM/SPF/DMARC, but it also needs security extensions in the DNS protocol. We completed our initial investigation of a DNSSEC implementation prototype, +and we discovered several issues, like some of the elliptic curve cryptography was missing. Those necessary cryptographic primitives are now available, so we +should complete this milestone by the end of the month.

Finally, our fourth milestone was to implement the Matrix protocol (client and server). We completed the protocol’s client library, which sends a notification +from OCaml CI. Plus, we have a PoC, and Matrix’s server-side, which received the notification, is also complete.

Although we still have much work ahead of us, we’re quite pleased with the progress thus far, and so is the DAPSI Initiative! Follow our progress by subscribing +to this blog and our Twitter feed (@tarides_) for the latest updates.

+
+

+ + + + +

The New Replaying Benchmark in Irmin

tarides — 2021-10-04T00:00:00-00:00

As mentioned in our Tezos Storage / Irmin Summer 2021 Update on the Tezos Agora forum, the Irmin team's goal has been to improve Irmin's performance in order to speed up the Baking Account migration process in Octez, and we managed to make it 10x faster in the first quarter of 2021. Since then, we've been working on a new benchmark program for Irmin that's based on the interactions between Irmin and Octez. This won't just help make Irmin even faster, it will also help speed up the Tezos blockchain process and enable us to monitor Irmin's behavior in Octez.

Octez is the Tezos node implementation that uses Irmin to store the blockchain state, so Irmin is a core component of Octez that's responsible for virtually all the filesystem operations. Whether a node is launched to produce new blocks (aka “bake”) or just to participate in peer-to-peer sharing of existing blocks, it must first update itself by rebuilding blocks individually until it reaches the head of the blockchain. This first phase is called bootstrapping, and once it reaches the blockchain head, we say it has been bootstrapped. Currently, the bootstrapped phase processes 2 block per minute, which is the rate at which the Tezos blockchain progresses. The next goal is to increase that rate to 12 blocks per minute.

Irmin stores the content of the Tezos blockchain on a disk using the irmin-pack library. There is one-to-one correspondence between the Tezos block and the Irmin commits. Each time Tezos produces a block, Irmin produces a commit, and then the Tezos block hash is computed using the Irmin commit hash. The Irmin developers are working on improving the irmin-pack performance which in turn will improve the performance of Octez.

A benchmark program is considered “fair” when it's representative of how the benchmarked code is used in the real world—for example, the access-patterns to Irmin. A standard database benchmark would first insert random data and then remove it. Such a synthetic benchmark would fail to reproduce the bottlenecks that occur when the insertions and removal are interleaved. Our solution to “fairness” is radical: replaying. Within a sandboxed environment, we replay a real world situation.

Basically, our new benchmark program makes use of a benchmarked code and records statistics for later analysis. The program is stored in the irmin-bench library and makes use of operation traces (called action traces) when Octez runs with Irmin. Later, the program replays the recorded operations one at a time while simultaneously recording tonnes of statistics (called stat traces). Data analysis of the stat traces may reveal many interesting facts about the behaviour of Irmin, especially if we tweak:

the configuration of Irmin (e.g., what’s the impact of doubling the size of a certain cache?)
the replay parameters (e.g., does Irmin's performance decay over time? Does irmin-pack perform as well after 24 hours of replay as after 1 minute of replay?)
the hardware (e.g., does irmin-pack perform well on a Raspberry Pi?)
the code of Irmin (e.g., does this PR have an impact on performance?)

This benchmarking process is similar to the record-replay feature available with TezEdge.

Recording the Action Trace

By adding logs to Tezos, we can record the Tezos-Irmin interactions and thus capture the Irmin “view” of Tezos. We’ve recorded action traces during the bootstrapping phase of Tezos nodes, which started from Genesis—the name of the very first Tezos block inserted into an empty Irmin store.

The interaction surface between Irmin and Octez is quite simple, so we were able to reduce it to eight (8) elementary operations:

checkout, to pull an Irmin tree from disk;
find, mem and mem_tree, read only operations on an Irmin tree;
add, remove and copy, write only operations on an Irmin tree;
commit, to push an Irmin tree to disk.

It’s important to remember that Irmin behaves much like Git. It has built-in snapshotting and is compatible with Git itself when using the irmin-git library. In fact, these operations are very similar to Git, too.

Sequence of Operations

To illustrate further, here's a concrete example of an operation sequence inside an action trace:

+ + + + +

This shows Octez’s first interaction with Irmin at the very beginning of the blockchain! The first block, Genesis, is quite small (it ends at operation #5), but the second one is massive (it ends at operation #309273). It contains no transactions because it only sets up the entire structure of the tree. It precedes the beginning of Tezos's initial protocol called “Alpha I”.

Benchmark Benefits

Our benchmark results convey the sheer magnitude of the Tezos blockchain and the role that Irmin plays within it. We’ve recorded a trace that covers the blocks from the beginning the blockchain in June 2018 all the way up to May 2021. It weighs 96GB.

Although it took 34 months for Tezos to reach that state, bootstrapping so far takes only 170 hours, and replaying it takes a mere 37 hours on a section of the blockchain that contains 1,343,486 blocks. On average, this corresponds to 1 per minute when the blocks were created, 132 per minute when bootstrapping, and 611 per minute during replay.

On this particular section of the blockchain, Octez had 1,089,853,521 interactions with Irmin. On average, this corresponds to 12 per second when the blocks were created, 1782 per second during bootstrapping, and 8258 per second during replay.

The chart below demonstrates how many of each Irmin operation occur per block (on average):

+ +

+ + + + +

+ +

This next chart displays where the time is spent during replay:

+ +

+ + + + +

+ +

With irmin-pack, an OCaml thread managed by the index library is running concurrently to the main thread (i.e., the merge thread), a fraction of the durations (shown above) are actually spent in that thread. Refer to this blog post for more details on index's merges.

The following chart illustrates how memory usage evolves during replay:

+ +

+ + + + +

+ +

On a logarithmic scale, this last chart shows the evolution of the write amplification, which indicates the amount of rewriting (e.g., at the end of the replay, 20TB of data have been written to disk in order to create a store that weighs 73GB).

+ +

+ + + + +

+ +

The merge operations of the index library are the source of this poor write amplification. The Irmin team is working hard on improving this metric:

on the one hand, the new structured keys feature of the upcoming Irmin 3.0 release will help to reduce the pressure on the index library,
on the other hand, we are working on algorithmic improvements of index itself.

Another nice way to use the trace is for testing. When replaying a trace, we can recompute the commit hashes and check that they correspond to the trace hashes, so the benchmark acts as additional tests to ensure we don't compromise the hashes computed in Tezos.

Complex changes to Tezos can be simulated first in Irmin. For example, the path flattening in Tezos feature (merged in August 2021) can now be tested earlier in the process with our benchmark. Prior to the trace benchmarks, we first had to make the changes in Tezos to understand their repercussions on Irmin directly from the Tezos benchmarks.

Lastly, we continue to test alternative libraries and compare them with the ones integrated in Tezos; however, using these alternative libraries to build Tezos nodes has proven to be more complicated than merely adding them in Irmin and running our benchmarks. While testing continues on most new libraries, we can definitely use replays to compare our new cactus library as a replacement for our index library.

Future Directions

While the action trace recording was only made possible on a development branch of Octez, we would next like to upstream the feature to the main branch of Octez, which would give all users the option to record Tezos-Irmin interactions. This would simplify bug reporting overall.

Although the first version only deals with the bootstapping phase of Tezos, an upcoming goal is to make it possible to benchmark the boostrapped phase of Tezos as well. Additionally, we plan to replay the multiprocess aspects of a Tezos node in the near future.

The first stable version of this benchmark has existed in Irmin’s development branch since Q2 2021, and we will release it as part of irmin-bench for Irmin 3.0 in Q4 2021. This release will allow integration into the Sandmark OCaml benchmarking suite.

Follow the Tarides blog for future Irmin updates.

Announcing Tezos’ 8th protocol upgrade proposal: Hangzhou

tarides — 2021-09-21T00:00:00-00:00

The last upgrade of the Tezos protocol, Granada, activated on August 6th, 2021. +We are now glad to announce a new protocol proposal, Hangzhou, the result of a +collaborative work from various teams.

This is a joint post with Nomadic Labs, +Marigold, Oxhead Alpha +and DaiLambda.

Tarides Returns to FIC 2021

tarides — 2021-09-06T00:00:00-00:00

Last year, Tarides had the honour of winning the “Coup de Coeur” Startup Award +at the International Cybersecurity Forum (FIC). It’s the leading cybersecurity +event in the EU. It’s both a forum, to present and discuss innovations and +reflect on the state of the European cybersecurity ecosystem, and a trade fair, +where cybersecurity and other tech professionals can meet and network.

This year, Tarides returns to FIC with their own booth! Our representatives, +including founder and CEO of Tarides, Thomas Gazagnaire, look forward to making +new connections, looking for collaborators, and catching up with colleagues.

The FIC theme for 2021 centres on encouraging “a collective and collaborative +cybersecurity.” From the FIC +Website:

+
”Collective, because each stakeholder is responsible not only for its own +security but also for the security of every other stakeholder, and therefore of +the whole. Collaborative, because cooperation and information sharing are +essential to compensate the asymmetry between the «attacker» and the +«defender»…FIC 2021 will focus on the major operational, industrial, +technological, and strategic challenges of cooperation”
+

Solutions for most cybersecurity issues already exist, and we at Tarides are +happy to consult on our many solutions that can make your systems more secure. +From secure emails to runtime protection to secure IoT protocols, our +representatives can help!

So stop by the Tarides booth at FIC to chat about your options. We’ll have +snacks and other goodies, too!

Benchmarking OCaml projects with current-bench

tarides — 2021-08-26T00:00:00-00:00

Regular CI systems are optimised for workloads that do not require stable performance over time. This makes them unsuitable for running performance benchmarks.

current-bench provides a predictable environment for performance benchmarks and a UI for analysing results over time. Similar to a CI system, it runs on pull requests and branches which allows performance to be analysed and compared. It can currently be enabled as an app on GitHub repositories with zero configuration. Several public repositories are runningcurrent-bench, including Irmin and Dune. We plan to enable it on more projects in the future.

In this article, we give a technical overview of current-bench, showing how results are collected and analysed, requirements for using it and how we built the infrastructure for stable benchmarks. We also describe future work that would allow more OCaml projects to run current-bench.

Introduction

For performance critical software, we must run benchmarks to ensure that there's no regression. Running benchmarks before the user submits their pull request is tedious, and since every user might have a different machine, you can't be sure if the benchmarks performed actually improved or regressed performance.

Our current-bench aims to solve this problem by providing a stable benchmarking platform that runs every time the user submits a pull request and compares the result to the benchmarks on the main branch. As current-bench is zero-configuration, users can enroll their repository to run benchmarks with ease. This current-bench has helped projects ensure that regression doesn't happen, so you can merge code with more confidence.

Architecture

+ + + + +

Benchmarking Pipeline

As shown in Figure 1 (above), the benchmarking infrastructure uses ocurrent¹, an embedded Domain Specific Language to write a pipeline. The ocurrent command computes the build incrementally and helps with static analysis. Whenever a pull request is opened on a repository monitored by current-bench, a POST request is sent to the server running the pipeline. The pipeline fetches the head commit on the pull request and uses Docker to compile the code, and then it runs the make bench command inside the generated Docker image.

The pipeline runs on a single node, and the process is pinned to a single core to ensure there's no contention of resources when running the benchmarks. Once finished, the raw JSON result is stored in a Postgres database, which the frontend can query using a GraphQL API, as shown in Figure 2 below.

+ + + + +

The frontend supports historical navigation and provides comparison with the default branch. It allows users to select a pull request of which they want to see the graphs. The graphs display the individual result of the head commit and the comparison with the commits on the default branch. The frontend permits users to select the historical interval when they want to compare benchmarks, and it also shows the standard deviation. Once the benchmarks have run successfully, the pipeline sets the pull request status to the frontend URL. Then the user can look at the graphs.

Hardware Optimisation

Our current-bench uses the hardware optimisations developed for OCaml multicore compiler benchmarks (presented at ICFP OCaml Workshop 2019) with a few modifications to allow the benchmarks to run inside Docker containers. To get stable performance, we configured the kernel to isolate some of the CPU cores. Linux then avoids scheduling other user processes automatically. We also disabled IRQ handling and power saving.

The container that runs the benchmark is pinned to one of the isolated cores. Since I/O operations can make the benchmarks less stable, we use an in-memory tmpfs partition in /dev/shm for all storage. For NUMA enabled systems, we configure this partition to be allocated on the NUMA node of the isolated core. The pipeline disables ASLR inside the container automatically, which is normally blocked by the default Docker seccomp profile, so we have modified the profile to allow the personality(2) syscall.

Enrolling a repository

To enroll a repository, you need to ensure the following:

Enable the ocaml-benchmarks GitHub app for your repository.
The repository needs a bench Makefile target. This is triggered from the current-bench pipeline.
The output of the make bench target is JSON, which can be parsed by the pipeline and displayed by the frontend.

Future work

Anyone who wants to roll out a continuous, zero-configured benchmarking infrastructure can set up the current-bench infrastructure. In the future, we want to scale current-bench by isolating cores on multiple machines and adding a scheduler to ensure that benchmarks use only one core at a time per machine. We plan to add support for different benchmarking libraries that repositories can use—for example, we currently support repositories using bechamel. We also aim to make the adoption of current-bench easier by adding a conversion library that can convert any benchmark output into output parseable by current-bench. We intend to add support for quick and slow benchmarks, which would allow users to have faster feedback loops on pull requests while ensuring they can still run more extensive, time consuming benchmarks to see the performance.

Thank you for reading! You can check out the implementation for current-bench here!

Tarides Engineers to Present at ICFP 2021

tarides — 2021-08-26T00:00:00-00:00

This year marks the 25th anniversary of the OCaml Language! It's an exciting +time for OCaml programmers and enthusiasts. A fun and informative way to +celebrate OCaml's birthday is to attend the 26th Annual International +Conference on Functional +Programming (ICFP), held online +this year due to ongoing Covid restrictions. While this is disappointing news +for so many, it's beneficial to those of you outside France because now you +can hear professionals talk about cutting edge technology from the comfort of +your own home.

Tarides engineers, as well as our colleagues at OCaml Labs +Consultancy and Segfault Systems, +have some exciting presentations at this year's ICFP! Listen to talks on running +OCaml on multiple cores, generating fuzzing suites, benchmarking, and the +experimental OCaml effects.

You can search the [complete ICFP +Timetable](https://icfp21.sigplan.org/program/program-icfp-2021/?past=Show +upcoming events only&date=Fri 27 Aug 2021) for other topics of interest and read +below about our engineers' projects and presentations. Times are listed both in +London (GMT +1) and Paris (GMT +2) for ease of planning. The following talks are +scheduled for Friday, 27 August 2021.

Grab a cup of coffee for our first morning talk at 9am London / 10am Paris +and learn about Adapting the OCaml Ecosystem for Multicore OCaml. With the +soon-to-be released OCaml 5.0, there will be support for Shared-Memory +Parallelism. There’s increasing interest in the community to port existing +libraries to Multicore, so this talk will cover the arrival of Multicore and +what that means to the OCaml ecosystem. Our engineers will highlight existing +tools and provide methods for a smooth transition, so viewers can benefit from +Multicore parallelism. They'll also share some insights from their experience +porting existing libraries to Multicore OCaml.

Read more about this topic on todays' post at Segfault +Systems, written by +one of tomorrow's presenters, Sudha +Parimala of Segfault Systems. +Joining Sudha for the presentation are Enguerrand +Decorne (Tarides), +Sadiq Jaffer (Opsian and OCaml +Labs Consultancy), Tom Kelly +(OCaml Labs Consultancy), and KC +Sivaramakrishnan of IIT +Madras.

Next up is Leveraging Formal Specifications to Generate Fuzzing Suites at +11:10 London / 12:10 Paris, presented by Tarides's own Nicolas +Osborne and Clément +Pascutto. They'll discuss +how developers typically first have to capture the semantics they want when +checking a library and then write the code implementing these tests and find +relevant test cases that expose possible misbehaviours. Through their work, +they'll present a tool that automatically takes care of those last two steps by +automatically generating fuzz testing suites from OCaml interfaces annotated +with formal behavioural specifications. They'll also show some ongoing +experiments on fuzzing capabilities and limitations applied to real-world +libraries.

Next up is our talk on Continuous Benchmarking for +OCaml Projects at 12:30 London / 13:30 Paris. Regular CI systems are +optimised for workloads that do not require stable performance over time, which +makes them unsuitable for running performance benchmarks. Tarides engineers +Gargi Sharma, Rizo +Isrof, and Magnus +Skjegstad will discuss how +current-bench provides a predictable environment for performance benchmarks +and a UI for analysing results over time. Similar to a CI system it runs on pull +requests and branches allowing performance to be analysed and compared, and it +can currently be enabled on as an app on GitHub repositories with zero +configuration. Several public repositories already run current-bench, +including Irmin and +Dune, and they plan to enable it on more +projects in the future. Read Gargi's recent blog post for more information on +benchmarking.

In this presentation, they will give a technical overview of current-bench, showing how results are collected and analysed, requirements for using it, and how they built the infrastructure for stable benchmarks. They'll also cover some future work that will allow more OCaml projects to run current-bench.

Immediately after the Benchmarking talk, catch A Multiverse of Glorious Documentation +scheduled at 12:50 London / 13:50 Paris. Lucas +Pluvinage of Tarides and +Jonathan Ludlam of OCaml +Labs Consultancy will discuss the process of generating documentation for every +version of every package that can be built from the Opam repository and present +it as a single coherent website that's continuously updated as new packages are +released and old packages are updated. They will address the challenges of +caching, handling different compiler versions, and incompatible libraries. The +process has been implemented as an OCurrent pipeline named ocaml-docs-ci and +is already available on Github. It has been used to produce the documentation of +more than 10,000 package versions, generating 2.5M HTML pages. That's 38GB of +artifacts!

After a relaxing lunch, come back for Experiences with Effects at 15:30 +London / 16:30 Paris. Join OCaml Labs and Tarides engineers Thomas +Leonard, Craig +Ferguson, Patrick +Ferris, Sadiq +Jaffer, Tom +Kelly, KC +Sivaramakrishnan, and +Anil Madhavapeddy as they +talk about an exciting, experimental branch of Multicore OCaml that adds support +for effect handlers. In this presentation, they'll discuss their experiences +with effects, both from converting existing code and from writing new code. They +discovered that converting the Angstrom parser from a callback style to effects +greatly simplified the code while also improving performance and reducing +allocations. Their experimental Eio +library uses effects that allows +writing concurrent code in direct style, without the need for monads (as found +in Lwt or Async).

Enjoy a full day of OCaml innovation and get to know some of our talented +engineers better by joining Tarides, OCaml Labs, and Segfault Systems at ICFP on +Friday, 27 August 2021. See you there!

Tarides at WomenHack Virtual Event

tarides — 2021-07-20T00:00:00-00:00

Tarides takes great pride in a diverse workforce and strives to continue +bringing talented people to its team from around the globe. This is why Sonja +Heinze, a Tarides software engineer, and the Head of HR, Héloïse Lutton, will +attend WomenHack, an online event dedicated to recruiting more women into the +tech world. They're participating not only to present Tarides to the Women In +Tech community, but to also network and possibly find new talented programmers +to join our growing team.

The WomenHack event has a unique setup similar to 'speed dating,' but for +prospective jobs. Each candidate is paired with a company, and they have 5 +minutes to chat before moving on to the next company. This "rapid interview" is +both fun and efficient for all involved, and it ensures each company will meet +several talented candidates from diverse backgrounds, which increases their +chance of finding that perfect fit for an open position!

From their website:

+
WomenHack is a community that empowers women in +tech through events, jobs, and reviews. We aim to create a more inclusive and +diverse workplace for all. Our diversity recruiting events target some of the +most talented women in tech which include software developers, designers, and +product talent.
+

Join us tomorrow, July 21st, at WomenHack! You can get your ticket +here.

Tarides Introduces OSMOSE at the Open-Source Innovation Sprint

tarides — 2021-06-29T00:00:00-00:00

Tarides is excited to announce that our CEO, Dr. Thomas Gazagnaire, and Prof. +Anil Madhavapeddy, from the University of Cambridge, will present their +innovative platform OSMOSE at the Open Source Innovation Sprint (OSIS) +conference on 1 July 2021. This event is organized by +Systematic.

OSMOSE is a software platform made to manage digital infrastructure at scale, +securely and efficiently. It uses the groundbreaking creation of unikernels to +radically simplify the way applications are built and deployed for the cloud.

Since digital transformation has become more and more dependent on cloud +computing, it has increasing problems with high response latency, security +risks, and resource inefficiencies—all which make these services ultimately +unreliable. The demand for interconnected devices is ever increasing, but the +security of these devices remain unchecked, making them susceptible to security +vulnerabilities. This leaves consumers and businesses open to exploitation, as +demonstrated in reports of tech devices violating users’ privacy, like sending +audio recordings without their knowledge or consent.

Tarides addresses these issues with OSMOSE, a platform which combines hardware +and software elements to invert the current cloud-centric model. OSMOSE securely +connects with physical spaces to provide extremely low latency and +high-bandwidth, local-area computation capabilities, which can turn a fleet of +IoT devices into a local data-centre.

This innovative platform enables computer resources to be tracked efficiently +and temporarily rented to users. This turns any IoT deployment into a local, +private cloud, allowing a better utilization of local resources and improved +security.

Major components of OSMOSE already have commercial applications. It’s been used +to make existing cloud deployments more secure and efficient by companies such +as Amazon, Citrix, and Docker.

Tarides applies a high-touch, mixed business strategy—using consultancy services +to field test open source components under development. Tarides applies their +research to real-world systems to build unikernels, a secure-by-design and +resource-efficient application specialised to their run-time environments. If +interested in using OSMOSE as a solution for your business, +please reach out to Tarides for more +information.

Tarides project SCoP is selected as one of the brightest Data Portability projects in Europe!

tarides — 2021-04-30T00:00:00-00:00

Tarides is taking part in the Data Portability & Services Incubator (DAPSI), a +3-year EU funded project that empowers internet innovators to develop new +solutions in the Data Portability field.

What is DAPSI?

The Data Portability and Services Incubator (DAPSI) is +an EU funded project, under the European Commission’s Next Generation Internet +(NGI) initiative. The aim of this initiave is to empower top internet innovators +to develop human-centric solutions. DAPSI addresses the challenge of personal +data portability on the internet, as foreseen under the GDPR and make it +significantly easier for citizens to have any data which is stored with one +service provider transmitted directly to another provider.

Take a look at the DAPSI innovators +portfolio to see more information about the +selected projects.

What is our project?

Our project, called SCoP for Secure-by-design Communication Protocols, +is taking part in the DAPSI to tackle data portability issues in communication +services.

Over the past few decades, the usage of emails has been massively widespread by +both individuals and companies. Billions of emails are sent every day and this +number is expected to increase to reach 333 billion of emails exchanged daily +in 2022. Moreover, as managing internet communication stacks have become +increasingly complex, end-users have tended to entrust this task to third-party +companies like Google and Microsoft. Furthermore, existing implementations of +these communication services rely on ad-hoc methodologies and memory-unsafe +languages, where minor developer errors can easily escalate into major security +flaws. The centralization of these communication services means that a single +successful attack leads to major personal data breaches.

To fix this issue, our project aims to engineer a modern basis for open +messaging that supports existing protocols such as emails but is also extensible +and customizable for emerging protocols such as matrix. We will be building +trustable implementations of these open protocols using type-safe languages and +we will deploy these implementations as specialized, secure and resource +efficient unikernels. They will become the basis of the communication system of +OSMOSE, Tarides’ commercial solution for secure-by-design IoT infrastructure.

Every component of that system will be carefully designed as independent +libraries, using modern development techniques to avoid the common reported +threats and flaws. For instance, the implementation of protocol parsers and +serializers will be written in a type-safe language and will be using fuzzing, +e.g state-of-the-art coverage-driven tests. The combination of these techniques +will increase users’ trust to migrate their personal data to these new secure +services.

Moreover, these techniques are also useful to produce a large and reusable +corpus of test materials, which we plan to release separately for other +implementations to use. It will give the tools to other developers to write the +next generation of messaging applications by extending the existing protocols +with more confidence.

Want to be part of it?

Would you like to hear more about the project? Or want to deploy our solution? +This project will build on a number of existing components in +MirageOS, such as +MrMime +and Irmin, +so feel free to contribute to these existing components! Please reach out to contact@tarides.com.

+
+

+ + + + +

Florence and beyond: the future of Tezos storage

tarides — 2021-03-04T00:00:00-00:00

In collaboration with Nomadic Labs, Marigold and DaiLambda, we're happy to +announce the completion of the next Tezos protocol proposal: +Florence.

Tezos is an open-source decentralised blockchain network providing a +platform for smart contracts and digital assets. A crucial feature of Tezos is +self-amendment: the network protocol can be upgraded +dynamically by the network participants themselves. This amendment process is +initiated when a participant makes a proposal, which is then subject to a +vote. After several years working on the Tezos storage stack, this is our first +contribution to a proposal; we hope that it will be the first of many!

As detailed in today's announcement from Nomadic Labs, +the Florence proposal contains several important changes, from the introduction +of Baking Accounts to major quality-of-life improvements for smart contract +developers. Of all of these changes, we're especially excited about the +introduction of sub-trees to the blockchain context API. In this post, we'll +give a brief tour of what these sub-trees will bring for the future of Tezos. +But first, what are they?

Merkle sub-trees

The Tezos protocol runs on top of a versioned tree called the “context”, which +holds the chain state (balances, contracts etc.). Ever since the pre-Alpha era, +the Tezos context has been implemented using Irmin – an open-source +Merkle tree database originally written for use by MirageOS unikernels.

For MirageOS, Irmin’s key strength is flexibility: it can run over arbitrary +backends. This is a perfect fit for Tezos, which must be agile and +widely-deployable. Indeed, the Tezos shell has already leveraged this agility +many times, all the way from initial prototypes using a Git backend to the +optimised irmin-pack implementation used today.

But Irmin can do more than just swapping backends! It also allows users to +manipulate the underlying Merkle tree structure of the store with a high-level +API. This “Tree” API enables lots of interesting use-cases of +Irmin, from mergeable data types (MRDTs) to zero-knowledge proofs. +Tezos doesn't use these more powerful features directly yet; that’s where Merkle +proofs come in!

Proofs and lightweight Tezos clients

Since the Tezos context keeps track of the current "state" of the blockchain, +each participant needs their own copy of the tree to run transactions against. +This context can grow to be very large, so it's important that it be stored as +compactly as possible: this goal shaped the design of irmin-pack, our latest +Irmin backend.

However, it's possible to reduce the storage requirements even further via the +magic of Merkle trees: individuals only need to store a fragment of the root +tree, provided they can demonstrate that this fragment is valid by sending +“proofs” of its membership to the other participants.

This property can be used to support ultra-lightweight Tezos clients, a feature +currently being developed by TweagIO. To make this a reality, +the Tezos protocol needs fine-grained access to context sub-trees in order build +Merkle proofs out of them. Fortunately, Irmin already supports this! We +extended the protocol to understand sub-trees, lifting the power +of Merkle trees to the user.

We’re excited to work with TweagIO and Nomadic Labs on lowering the barriers to +entering the Tezos ecosystem and look forward to seeing what they achieve with +sub-trees!

Efficient Merkle proof representations

Simply exposing sub-trees in the Tezos context API isn’t quite enough: +lightweight clients will also need to serialize them efficiently, since proofs +must be exchanged over the network to establish trust between collaborating +nodes. Enter Plebeia.

Plebeia is an alternative Tezos storage layer – developed by DaiLambda – with +strengths that complement those of Irmin. In particular, Plebeia is capable of +generating very compact Merkle proofs. This is partly due to its specialized +store structure, and partly due to clever optimizations such as path compression +and inlining.

We’re working with the DaiLambda team to unite the strengths of Irmin and +Plebeia, which will bring built-in Merkle proof support to the Tezos storage +stack. The future is bright for Merkle proofs in Tezos!

Baking account migrations

Trees don’t just enable new features; they have a big impact on performance +too! Currently, indexing into the context always happens from its root, which +duplicates effort when accessing adjacent values deep in the tree. Fortunately, +the new sub-trees provide a natural representation for “cursors” into the +context, allowing the protocol to optimize its interactions with the storage +layer.

To take just one example, DaiLambda recently exploited this feature to reduce +the migration time necessary to introduce Baking Accounts to the network by a +factor of 15! We’ll be teaming up with Nomadic Labs and DaiLambda to ensure that +Tezos extracts every bit of performance from its storage.

It's especially exciting to have access to lightning-fast storage migrations, +since this enables Tezos to evolve rapidly even as the ecosystem expands.

Storage in other languages

Of course, Tezos isn’t just an OCaml project: the storage layer also has a +performant Rust implementation as part of TezEdge. We’re working with +Simple Staking to bring Irmin to the Rust community via an +FFI toolchain, enabling closer alignment between the different +Tezos shell implementations.

Conclusion

All in all, it’s an exciting time to work on Tezos storage, with many +open-source collaborators from around the world. We’re especially happy to see +Tezos taking greater advantage of Irmin’s features, which will strengthen both +projects and help them grow together.

If all of this sounds interesting, you can play with it yourself using the +recently-released Irmin 2.5.0. Thanks for reading, and stay tuned for +future Tezos development updates!

Partnering for more diversity in Tech

tarides — 2021-02-15T00:00:00-00:00

Tarides is very glad to announce our partnership with Ada Tech +School.

Founded in 2019 and based in Paris (France), Ada Tech School, named for pioneer +computer scientist Ada Lovelace, +is a programming school designed for women but open to all. The program is +driven by three values: feminism, empathy and singularity. Its mission is to +facilitate access to programming positions and promote the feminization of tech, +by creating training that tackles the gender and cultural biases of IT.

Unfortunately, the diversity of the candidate pool is very limited when a +company tries to fill positions. Barely 10% of computer science students in +France are girls. Ada Tech School is an excellent initiative to democratize +software education amongst women. The school was created so that women can land +a job easily in the IT industry through rigorous training, and then offer +ongoing coaching and support to ascend the professional ladder within tech +companies.

At Tarides, we believe that a healthy team is a diverse one; and that trust, +fairness and inclusion are values needed to build a strong company.

We are committed to doing better, not only by hiring a diverse team and +providing a welcoming work environment, but also by putting people first at +every stage. This means providing fair and equitable compensation as well as +meaningful career advancement opportunities for every employee.

We believe that a great technology always derives from great people, regardless +of their background. Head here to see our +currently-open positions.

Recent and upcoming changes to Merlin

tarides — 2021-01-26T00:00:00-00:00

Merlin is a language server for the OCaml programming language; that is, a daemon +that connects to your favourite text editor and provides the usual services of +an IDE: instant feedback on warnings and errors, autocompletion, "type of the +code under the cursor", "go to definition", etc. As we (Frédéric Bour, Ulysse +Gérard and I) are about to do a new major release, we thought now would be a +good time to talk a bit about some of the changes that are going into this +release.

Project configuration

Since its very first release, merlin has been getting information about the +project being worked on through a .merlin file, which used to be written by +the user, but is now often generated by build systems.

This had the advantage of being fairly simple: Merlin would just look in the +current directory if such a file existed, otherwise it would look in the parent +directories until it found one; and then read it. But there were also some +sore points: the granularity of the configuration is the directory not the file, +and this information is duplicated from the build system configuration (be it +dune, Makefiles, or, back in the days, ocamlbuild).

After years of thinking about it, we've finally decided to make some light +changes to this process. Since version 3.4, when it scans the filesystem Merlin +is now looking for either a .merlin file or a dune (or dune-project) file. And +when it finds one of those, it starts an external process in the directory where +that file lives, and asks that process for the configuration of the ml(i) file +being edited.

The process in charge of communicating the configuration to Merlin will either +be a specific dune subcommand (when a dune file is found), or a dedicated +.merlin reader program.

We see several advantages in doing things this way (rather than, for instance, +changing the format of .merlin files):

this change is entirely backward compatible, and indeed the transition has +already happened silently; although dune is still emitting .merlin files, +this will only stop with dune 2.8.
externalizing the reading of .merlin files and simply requiring a +"normalized" version of the config (i.e. with no mention of packages, just of +flags and paths) allowed us to simplify the internals of Merlin.
talking to the build system directly not only gets us a much finer grained +configuration (which is important when you build different executables with +different flags in the same directory, or if you apply different ppxes to +different files of a library), it opens the door to getting a nicer behavior +of Merlin in some circumstances. For instance, the build system can (and +does) tell Merlin when the project isn't built. Currently we only report that +information to the user when he asks for errors, alongside all the other +(mostly rubbish) errors. Which is already helpful in itself. But in the +future we can start filtering the other errors to only report those that +would remain even after building the project (e.g. parse errors).

There are however some changes to look out for:

people who still use .merlin files but do not install Merlin using opam need +to make sure to also have the dot-merlin-reader binary in their PATH (it is +available as an opam package, but is also buildable from Merlin's git +repository)
vim and emacs users who could previously load packages interactively (by +calling M-x merlin-use or :MerlinUse) cannot do that anymore, since Merlin +itself stopped linking with findlib. They'll have to write a .merlin file.

Dropping support for old versions of OCaml

Until now, every release of Merlin has kept support from OCaml 4.02 to the +latest version of OCaml available at the time of that release.

We have done this by having one version of "the frontend" (i.e. handling of +buffer state, project configuration; analyses like jump-to-definition, +prefix-completion, etc.), but several versions of "the backend" (OCaml's +ASTs, parser and typechecker), and choosing at build time which one to use. +The reason for doing this instead of having, for instance, one branch of Merlin +per version of OCaml, is that while the backends are fairly stable once +released, Merlin's frontend keeps evolving. Having just one version of it makes +it easier to add features and fix bugs (patches don't need to be duplicated), +whilst ensuring that Merlin's behavior is consistent across every version of +OCaml that we support.

For this to work however, one needs a well defined API between the frontend and +all the versions of the backend. This implies mapping every versions of OCaml's +internal ASTs (which receive modifications from one version to the next), to a +unified one, so as to keep Merlin's various features version agnostic. But it +also means being resilient to OCaml's internal API changes. For instance between +4.02 and 4.11 there were big refactorings impacting: the way one accesses the +typing environment, the way one accesses the "load path" (the part of the file +system the compiler/Merlin is aware of), the way error message are produced, ...

The rate of changes on the compiler is a lot higher than what it was when we +first started Merlin (7 years ago now!) which doesn't just mean that we have to +spend more and more time on updating the common interface, but also that the +interface is getting harder to define. Recently (with the 4.11 release) some of +the changes were significant enough that for some parts of the backend we just +didn't manage to produce a single interface to access old and new versions, so +instead we had to start duplicating and specializing parts of the frontend. +And we don't expect things to get much better in the near future.

Furthermore, Merlin's backends are patched to be more resilient to parsing and +typing errors in the user's code. Those patches also need to be evolved at each +new release of the compiler. +The work required to keep the "unified interface" working was taking time away +from updating our patches properly, and our support of user errors has slowly +been getting worse over the past few years, resulting in less precise type +information when asked, incomplete results when asking for auto-completion, etc.

Therefore we have decided to stop dragging older versions of OCaml along. We +plan to switch to a system where we have one branch of Merlin per version of +OCaml, and each opam release of Merlin will only be buildable with one version +of OCaml. We will keep maintaining all the relatively recent branches (that is: +4.02 definitely will not get fixes, but 4.06 is still in the clear). However, +all the new features will be developed against the latest version of OCaml and +cherry-picked to older branches if, and only if, there are no merge conflicts +and they work as expected without changes.

We hope that this will make it easier for us to update to new versions of OCaml +(actually, we already know it does, working on adding support for 4.12 was +easier than for any of the other recent versions), will allow us to clean up +Merlin's codebase (let's call that a work in progress), and will free some time +to work on new features.

You might wonder what all this changes for you, as a user, in practice. Well, it +depends:

if you install Merlin from opam: nothing, or almost nothing. Everything that +you currently do with Merlin will keep working. In the future, perhaps some +new feature will appear that won't work on all versions. But that day hasn't +come yet.
if you install Merlin some other way (manually?): you can't just fetch master +and build it anymore. You have to pick the appropriate branch for your +version of OCaml.
if you're reusing Merlin's codebase as part of another project and (even +worse) have patches on it: come and talk to us if you haven't done so already! +We can try and integrate your patches, so that you only need to worry about +vendoring the right version(s) for your needs.

Over the years, Merlin has received bugfixes and improvements from a long list of +people, but for the upcoming release Frédéric and I are particularly grateful to +Rudi Grinberg, a long time and regular contributor who also maintains the OCaml +LSP project, as well as Ulysse Gérard, who joined our team a year ago now. They +are in particular the main authors of the work to improve the handling of +projects' configuration.

We hope you'll be as excited as us by all these changes!

Tarides sponsors the Oxbridge Women in Computer Science Conference 2020

tarides — 2020-12-14T00:00:00-00:00

The Oxbridge Women in Computer Science +conference is an annual one-day +event hosted by the Universities of Oxford and Cambridge (UK). The +conference is free and open to everyone from any discipline, regardless of +gender identity. Its purpose is to spotlight the successes of women within +computer science and strengthen the network of women in computer science +within a supportive and friendly environment.

This year, the conference was organised by the University of Cambridge and was +held virtually on December 7th.

Tarides is very glad to sponsor this event as we strongly believe that diversity +and inclusive culture is a key factor in building a competitive and innovative +company. Our employees come from 8 different countries and are 1⁄3 women. +Tarides promotes transparency, openness and autonomy, creating a work atmosphere +auspicious for employees to strive in their work, to solve novel, impactful and +technical challenges. By working on open-source projects, a collaboration is +possible with worldwide experts from both academia and industry, encouraging +continuous training and education; in this context, it is very important to have +teams with diverse backgrounds and experience.

The underrepresentation of women in tech, and particularly in computer science, +is not a new problem and gender equality remains a major issue in the corporate +world. By celebrating female computer scientists, as during the Oxbridge Women +in Computer Science Conference, it will hopefully encourage more women to pursue +their interests and careers in the tech field. Head +here to see our currently-open positions.

For the event, we made a short video to experience a day in the life of a +software engineer:

+ +

Building portable user interfaces with Nottui and Lwd

tarides — 2020-09-24T00:00:00-00:00

At Tarides, we build many tools and writing UI is usually a tedious task. In this post we will see how to write functional UIs in OCaml using the Nottui & Lwd libraries.

These libraries were developed for Citty, a frontend to the Continuous Integration service of OCaml Labs.

+ + + + +

In this recording, you can see the lists of repositories, branches and jobs monitored by the CI service, as well as the result of job execution. Most of the logic is asynchronous, with all the contents being received from the network in a non-blocking way.

Nottui extends Notty, a library for declaring terminal images, to better suit the needs of UIs. Lwd (Lightweight Document) exposes a simple form of reactive computation (values that evolve over time). It can be thought of as an alternative to the DOM, suitable for building interactive documents. +They are used in tandem: Nottui for rendering the UI and Lwd for making it interactive.

Nottui = Notty with layout and events

Notty exposes a nice way to display images in a terminal. A Notty image is matrix of characters with optional styling attributes (tweaking foreground and background colors, using bold glyphs...).

These images are pure values and can be composed (concatenated, cropped, ...) very efficiently, making them very convenient to manipulate in a functional way.

However these images are inert: their contents are fixed and their only purpose is to be displayed. Nottui reuses Notty images and exposes essentially the same interface but it adds two features: layout & event dispatch. UI elements now adapt to the space available and can react to keyboard and mouse actions.

Layout DSL. Specifying a layout is done using "stretchable" dimensions, a concept loosely borrowed from TeX. Each UI element has a fixed size (expressed as a number of columns and rows) and a stretchable size (possibly empty). The stretchable part is interpreted as a strength that is used to determine how to share the space available among all UI elements.

This is a simple system amenable to an efficient implementation while being powerful enough to express common layout patterns.

Event dispatch. Reacting to mouse and keyboard events is better done using local behaviors, specific to an element. In Nottui, images are augmented with handlers for common actions. There is also a global notion of focus to determine which element should consume input events.

Interactivity with Lwd

Nottui's additions are nice for resizing and attaching behaviors to images, but they are still static objects. In practice, user interfaces are very dynamic: parts can be independently updated to display new information.

This interactivity layer is brought by Lwd and is developed separately from the core UI library. It is built around a central type, 'a Lwd.t, that represents a value of type 'a that can change over time.

Lwd.t is an applicative functor (and even a monad), making it a highly composable abstraction.

Primitive changes are introduced by Lwd.var, which are OCaml references with an extra operation val get : 'a Lwd.var -> 'a Lwd.t. This operation turns a variable into a changing value that changes whenever the variable is set.

In practice this leads to a mostly declarative style of programming interactive documents (as opposed to the DOM that is deeply mutable). Most of the code is just function applications without spooky action at a distance! However, it is possible to opt-out of this pure style by introducing an Lwd.var, on a case-by-case basis.

And much more...

A few extra libraries are provided to target more specific problems.

Lwd_table and Lwd_seq are two datastructures to manipulate dynamic collections. Nottui_pretty is an interactive pretty printing library that supports arbitrary Nottui layouts and widgets. Finally Tyxml_lwd is a strongly-typed abstraction of the DOM driven by Lwd.

Version 0.1 has just been released on OPAM.

Getting started!

Here is a small example to start using the library. First, install the Nottui library:

$ opam install nottui

Now we can play in the top-level. We will start with a simple button that counts the number of clicks:

$ utop
+# #require "nottui";;
+# open Nottui;;
+# module W = Nottui_widgets;;
+(* State for holding the number of clicks *)
+# let vcount = Lwd.var 0;;
+(* Image of the button parametrized by the number of clicks *)
+# let button count =
+    W.button ~attr:Notty.A.(bg green ++ fg black)
+      (Printf.sprintf "Clicked %d times!" count)
+      (fun () -> Lwd.set vcount (count + 1));;
+(* Run the UI! *)
+# Ui_loop.run (Lwd.map button (Lwd.get vcount));;

Note: to quit the example, you can press Ctrl-Q or Esc.

We will improve the example and turn it into a mini cookie clicker game.

(* Achievements to unlock in the cookie clicker *)
+# let badges = [15, "Cursor"; 50, "Grandma"; 150, "Farm"; 300, "Mine"];;
+(* List the achievements unlocked by the player *)
+# let unlocked_ui count =
+    (* Filter the achievements *)
+    let predicate (target, text) =
+      if count >= target
+      then Some (W.printf "% 4d: %s" target text)
+      else None
+    in
+    (* Concatenate the UI elements vertically *)
+    Ui.vcat (List.filter_map predicate badges);;
+(* Display the next achievement to reach *)
+# let next_ui count =
+    let predicate (target, _) = target > ciybt in
+    match List.find_opt predicate badges with
+    | Some (target, _) ->
+      W.printf ~attr:Notty.A.(st bold) "% 4d: ???" target
+    | None -> Ui.empty;;
+(* Let's make use of the fancy let-operators recently added to OCaml *)
+# open Lwd_infix;;
+# let ui =
+    let$ count = Lwd.get vcount in
+    Ui.vcat [button count; unlocked_ui count; next_ui count];;
+(* Launch the game! *)
+# Ui_loop.run ui;;

+ + + + +

Et voilà! We hope you enjoy experimenting with Nottui and Lwd. Check out the Nottui page for more examples, and watch our recent presentation of these libraries at the 2020 ML Workshop here:

+ + +

Tarides is now a sponsor of the OCaml Software Foundation

tarides — 2020-09-17T00:00:00-00:00

Tarides is pleased to provide support for the OCaml Software +Foundation, a non-profit foundation hosted by +the Inria Foundation. The OCaml Software Foundation's mission is to +promote the OCaml programming language and its ecosystem by +supporting the growth of a diverse and international community of +OCaml users.

Tarides develops secure-by-design solutions in which OCaml's memory and +type-safety guarantees play a major role. Hence, most of the software +development that is done at Tarides is in OCaml: for instance, +MirageOS, a library operating system that +constructs unikernels for secure, high-performance network +applications; and Irmin, a library for building +mergeable, branchable distributed data stores, with built-in +snapshotting and support for a wide variety of storage backends.

Tarides is also very involved in the OCaml compiler development and +OCaml developer tooling ecosystem: as active maintainers of the OCaml +platform, Tarides is involved with most of the major +OCaml developer tools, including opam, dune and merlin.

Irmin: September 2020 update

tarides — 2020-09-08T00:00:00-00:00

This post will survey the latest design decisions and performance improvements +made to irmin-pack, the Irmin storage backend used by +Tezos. Tezos is an open-source blockchain technology, +written in OCaml, which uses many libraries from the MirageOS ecosystem. For +more context on the design of irmin-pack and how it is optimised for the Tezos +use-case, you can check out our previous blog post.

This post showcases the improvements to irmin-pack since its initial +deployment on Tezos:

Faster read-only store instances

The Tezos use-case of Irmin requires both read-only and read-write +store handles, with multiple readers and a single writer all accessing the same +Irmin store concurrently. These store handles are held by different processes +(with disjoint memory spaces) so the instances must use files on disk to +synchronise, ensuring that the readers never miss updates from the writer. The +writer instance automatically flushes its internal buffers to disk at regular +intervals, allowing the readers to regularly pick up replace calls.

Until recently, each time a reader looked for a value – be it a commit, a node, +or a blob – it first checked if the writer had flushed new contents to disk. This +ensured that the readers always see the latest changes from the writer. However, +if the writer isn't actively modifying the regions being read, the readers make +one unnecessary system call per find. The higher the rate of reads, the more +time is lost to these synchronisation points. This is particularly problematic +in two use-cases:

+
Taking snapshots of the store. Tezos supports exporting portable +snapshots of the store data. Since this operation only reads +historic data in the store (traversing backwards from a given block hash), +it's never necessary to synchronise with the writer.
+
+
Bulk writes. It's sometimes necessary for the writer to dump lots of new +data to disk at once (for instance, when adding a commit to the history). In +these cases, any readers will repeatedly synchronise with the disk even though +they don't need to do so until the bulk operation is complete. More on this in +the coming months!
+

To better support these use-cases, we dropped the requirement for readers to +maintain strict consistency with the writer instance. Instead, readers can call +an explicit sync function only when they need to see the latest concurrent +updates from the writer instance.

In our benchmarks, there is a clear speed-up for find operations from readers:

[RO] Find in random order with implicit syncs
+        Total time: 67.276527
+        Operations per second: 148640.253086
+        Mbytes per second: 6.378948
+        Read amplification in syscalls: 3.919739
+        Read amplification in bytes: 63.943734
+
+[RO] Find in random order with only one call to sync
+        Total time: 40.817458
+        Operations per second: 244993.208543
+        Mbytes per second: 10.513968
+        Read amplification in syscalls: 0.919588
+        Read amplification in bytes: 63.258072

Not only it is faster, we can see also that fewer system calls are used in the +Read amplification in syscalls column. The benchmarks consists of reading +10,000,000 entries of 45 bytes each.

Relevant PRs: irmin #1008, +index #175, +index #198 and +index #203.

Better flushing for the read-write instance

Irmin-pack uses an index to speed up find +calls: a pack file is used to store pairs of (key, value) and an index +records the address in pack where a key is stored. A read-write instance has +to write both the index and the pack file, for a read-only instance to find +a value. Moreover, the order in which the data is flushed to disk for the two +files is important: the address for the pair (key, value) cannot be written +before the pair itself. Otherwise the read-only instance can read an address for +a non existing (key, value) pair. But both pack and index have internal +buffers that accumulate data, in order to reduce the number of system calls, and +both decide arbitrarily when to flush those buffers to disk.

We introduce a flush_callback argument in index, which registers a callback +for whenever the index decides to flush. irmin-pack uses this callback to flush +its pack file, resolving the issue of the dangling address.

Relevant PRs: index #189, +index #216, +irmin #1051.

Faster serialisation for `Irmin.Type`

Irmin uses a library of generic operations: functions +that take a runtime representation of a type and derive some operation on that +type. These are used in many places to automatically derive encoders and +decoders for our types, which are then used to move data to and from disk. For +instance:

val decode : 'a t -> string -> 'a
+(** [decode t] is the binary decoder of values represented by [t]. *)
+
+(** Read an integer from a binary-encoded file. *)
+let int_of_file ~path = open_in_bin path |> input_line |> decode Irmin.Type.int32

The generic decode takes a representation of the type int32 and uses +this to select the right binary decoder. Unfortunately, we pay the cost of this +runtime specialisation every time we call int_of_file. If we're invoking +the decoder for a particular type very often – such as when serialising store +values – it's more efficient to specialise decode once:

(** Specialised binary decoder for integers. *)
+let decode_int32 = decode Irmin.Type.int32
+
+let int_of_file_fast ~path = open_in_bin path |> input_line |> decode_int32 contents

The question then becomes: how can we change decode to encourage it to be +used in this more-efficient way? We can add a type wrapper – called staged – +to prevent the user from passing two arguments to decode at once:

module Staged : sig
+  type +'a t
+  val   stage : 'a   -> 'a t
+  val unstage : 'a t -> 'a
+end
+
+val decode : 'a t -> (string -> 'a) Staged.t
+(** [decode t] needs to be explicitly unstaged before being used. *)

By forcing the user to add a Staged.unstage type coercion when using this +function, we're encouraging them to hoist such operations out of their +hot-loops:

(** The slow implementation no longer type-checks: *)
+
+let int_of_file ~path = open_in_bin path |> input_line |> decode Irmin.Type.int32
+(* Error: This expression has type (string -> 'a) Staged.t
+ *        but an expression was expected of type string -> 'a *)
+
+(* Instead, we know to pull [Staged.t] values out of hot-loops: *)
+
+let decode_int32 = Staged.unstage (decode Irmin.Type.int32)
+
+let int_of_file_fast ~path = open_in_bin path |> input_line |> decode_int32 contents

We made similar changes to the performance-critical generic functions in +Irmin.Type, and observed significant performance improvements. +We also added benchmarks for serialising various types.

+ +

Relevant PRs: irmin #1030 and +irmin #1028.

There are other interesting factors at play, such as altering decode to +increase the efficiency of the specialised decoders; we leave this for a future +blog post.

More control over `Index.merge`

index regularly does a maintenance operation, called merge, to ensure fast +look-ups while having a small memory imprint. This operation is concurrent with +the most of other functions, it is however not concurrent with itself: a second +merge needs to wait for a previous one to finish. When writing big chunks of +data very often, merge operations become blocking. To help measuring and +detecting a blocking merge, we added in the index API calls to check whether +a merge is ongoing, and to time it.

We mentioned that merge is concurrent with most of the other function in +index. One notable exception was close, which had to wait for any ongoing +merge to finish, before closing the index. Now close interrupts an ongoing +merge, but still leaves the index in a clean state.

Relevant PRs: index #185, +irmin #1049 and +index #215.

Clearing stores

Another feature we recently added is the possibility to clear the store. It is +implemented by removing the old files on disk and opening fresh ones. However +in irmin-pack, the read-only instance has to detect that a clear occurred. To +do this, we add a generation in the header of the files used by an +irmin-pack store, which is increased by the clear operation. A generation +change signals to the read-only instance that it needs to close the file and +open it again, to be able to read the latest values.

As the header of the files on disk changed with the addition of the clear +operation, the irmin-pack stores created previous to this change are no longer +supported. We added a migration function for stores created with the previous +version (version 1) to the new version (version 2) of the store. You can call +this migration function as follows:

 let open_store () =
+    Store.Repo.v config
+  in
+  Lwt.catch open_store (function
+      | Irmin_pack.Unsupported_version `V1 ->
+          Logs.app (fun l -> l "migrating store to version 2") ;
+          Store.migrate config ;
+          Logs.app (fun l -> l "migration ended, opening store") ;
+          open_store ()
+      | exn ->
+          Lwt.fail exn)

Relevant PRs: index #211, +irmin #1047, +irmin #1070 and +irmin #1071.

Conclusion

We hope you've enjoyed this discussion of our recent work. Stay +tuned for our next Tezos / MirageOS development update! Thanks +to our commercial customers, users and open-source contributors for making this +work possible.

Introducing irmin-pack

tarides — 2020-09-01T00:00:00-00:00

irmin-pack is an Irmin storage backend +that we developed over the last year specifically to meet the +Tezos use-case. Tezos nodes were initially using an +LMDB-based backend for their storage, which after only a year of activity led to +250 GB disk space usage, with a monthly growth of 25 GB. Our goal was to +dramatically reduce this disk space usage.

Part of the Irmin.2.0.0 release +and still under active development, it has been successfully integrated as the +storage layer of Tezos nodes and has been running in production for the last ten +months with great results. It reduces disk usage by a factor of 10, while still +ensuring similar performance and consistency guarantees in a memory-constrained +and concurrent environment.

irmin-pack was presented along with Irmin v2 at the OCaml workshop 2020; you +can watch the presentation here:

+ + +

General structure

irmin-pack exposes functors that allow the user to provide arbitrary low-level +modules for handling I/O, and provides a fast key-value store interface composed +of three components:

The pack is used to store the data contained in the Irmin store, as blobs.
The dict stores the paths where these blobs should live.
The index keeps track of the blobs that are present in the repository by +containing location information in the pack.

Each of these use both on-disk storage for persistence and concurrence and +various in-memory caches for speed.

Storing the data in the `pack` file

The pack contains most of the data stored in this Irmin backend. It is an +append-only file containing the serialized data stored in the Irmin repository. +All three Irmin stores (see our architecture +page in the tutorial to learn more) +are contained in this single file.

Content and Commit serialization is straightforward through +Irmin.Type. They are written along with their length (to allow +correct reading) and hash (to enable integrity checks). The hash is used to +resolve internal links inside the pack when nodes are written.

+ + + + +

Optimizing large nodes

Serializing nodes is not as simple as contents. In fact, nodes might contain an +arbitrarily large number of children, and serializing them as a long list of +references might harm performance, as that means loading and writing a large +amount of data for each modification, no matter how small this modification +might be. Similarly, browsing the tree means reading large blocks of data, even +though only one child is needed.

For this reason, we implemented a Patricia Tree representation of +internal nodes that allows us to split the child list into smaller parts that +can be accessed and modified independently, while still being quickly available +when needed. This reduces duplication of tree data in the Irmin store and +improves disk access times.

Of course, we provide a custom hashing mechanism, so that hashing the nodes +using this partitioning is still backwards-compatible for users who rely on hash +information regardless of whether the node is split or not.

Optimizing internal references

In the Git model, all data are content-addressable (i.e. data are always +referenced by their hash). This naturally lends to indexing data by hashes on +the disk itself (i.e. the links from commits to nodes and from nodes to +nodes or contents are realized by hash).

We did not comply to this approach in irmin-pack, for at least two reasons:

+
Referencing by hash does not allow fast recovery of the children, since +there is no way to find the relevant blob directly in the pack by providing +the hash. We will go into the details of this later in this post.
+
+
While hashes are being used as simple objects, their size is not negligible. +The default hashing function in Irmin is BLAKE2B, which provides 32-byte +digests.
+

Instead, our internal links in the pack file are concretized by the offsets – +int64 integers – of the children instead of their hash. Provided that the +trees are always written bottom-up (so that children already exist in the pack +when their parents are written), this solves both issues above. The data handled +by the backend is always immutable, and the file is append-only, ensuring that +the links can never be broken.

Of course, that encoding does not break the content-addressable property: one +can always retrieve an arbitrary piece of data through its hash, but it allows +internal links to avoid that indirection.

Deduplicating the path names through the `dict`

In fact, the most common operations when using irmin-pack consist of modifying +the tree's leaves rather then its shape. This is similar to the way most of us +use Git: modifying the contents of files is very frequent, while renaming or +adding new files is rather rare. Even still, when writing a node in a new +commit, that node must contain the path names of its children, which end up +being duplicated a large number of times.

The dict is used for deduplication of path names so that the pack file can +uniquely reference them using shorter identifiers. It is composed of an +in-memory bidirectional hash table, allowing to query from path to identifier +when serializing and referencing, and from identifier to path when deserializing +and dereferencing.

To ensure persistence of the data across multiple runs and in case of crashes, +the small size of the dict – less than 15 Mb in the Tezos use-case – allows +us to write the bindings to a write-only, append-only file that is fully read +and loaded on start-up.

We guarantee that the dict memory usage is bounded by providing a capacity +parameter. Adding a binding is guarded by this capacity, and will be inlined in +the pack file in case this limit has been reached. This scenario does not +happen during normal use of irmin-pack, but prevents attacks that would make +the memory grow in an unbounded way.

+ + + + +

Retrieve the data in the `pack` by indexing

Since the pack file is append-only, naively reading its data would require a +linear search through the whole file for each lookup. Instead, we provide an +index that maps hashes of data blocks to their location in the pack file, +along with their length. This module allows quick recovery of the values queried +by hash.

It provides a simple key-value interface, that actually hides the most complex +part of irmin-pack.

type t
+val v : readonly:bool -> path:string -> t
+
+val find    : t -> Key.t -> Value.t
+val replace : t -> Key.t -> Value.t -> unit
+(* ... *)

It has lead most of our efforts in the development of irmin-pack and is now +available as a separate library, wisely named index, that you can checkout on +GitHub under mirage/index and via opam as +the index and index-unix packages.

When index is used inside irmin-pack, the keys are the hashes of the data +stored in the backend, and the values are the (offset, length) pair that +indicates the location in the pack file. From now on in this post, we will +stick to the index abstraction: key and value will refer to the keys and +values as viewed by the index.

Our index is split into two major parts. The log is relatively small, and most +importantly, bounded; it contains the recently-added bindings. The data is +much larger, and contains older bindings.

The log part consists of a hash table associating keys to values. In order to +ensure concurrent access, and to be able to recover on a crash, we also maintain +a write-only, append-only file with the same contents, such that both always +contain exactly the same data at any time.

When a new key-value binding is added index, the value is simply serialized +along with its key and added to the log.

An obvious caveat of this approach is that the in-memory representation of the +log (the hashtable) is unbounded. It also grows a lot, as the Tezos node +stores more that 400 million objects. Our memory constraint obviously does not +allow such unbounded structures. This is where the data part comes in.

When the log size reaches a – customizable – threshold, its bindings are +flushed into a data component, that may already contain flushed data from +former log overloads. We call this operation a merge.

+ + + + +

The important invariant maintained by the merge operation is that the data +file must remain sorted by the hash of the bindings. This will enable a fast +recovery of the data.

During this operation, both the log and the former data are read in sorted +order – data is already sorted, and log is small thus easy to sort in +memory – and merged into a merging_data file. This file is atomically renamed +at the end of the operation to replace the older data while still ensuring +correct concurrent accesses.

This operation obviously needs to re-write the whole index, so its execution +is very expensive. For this reason, it is performed by a separate thread in the +background to still allow regular use of the index and be transparent to the +user.

In the meantime, a log_async – similar to log, with a file and a hash table +– is used to hold new bindings and ensure the data being merged and the new data +are correctly separated. At the end of the merge, the log_async becomes the +new log and is cleared to be ready for the next merge.

Recovering the data

This design allows us a fast lookup of the data present in the index. Whenever +find or mem is called, we first look into the log, which is simply a call +to the corresponding Hashtbl function, since this data is contained in memory. +If the data is not found in the log, the data file will be browsed. This +means access to recent values is generally faster, because it does not require +any access to the disk.

Searching in the data file is made efficient by the invariant that we kept +during the merge: the file is sorted by hash. The search algorithm consists in +an interpolation search, which is permitted by the even distribution of the +hashes that we store. The theoretical complexity of the interpolation search is +O(log (log n)), which is generally better than a binary search, provided that +the computation of the interpolant is cheaper than reads, which is the case +here.

This approach allows us to find the data using approximately 5-6 reading steps +in the file, which is good, but still a source of slowdowns. For this reason, we +use a fan-out module on top of the interpolation search, able to tell us the +exact page in which a given key is located, in constant time, for an additional +space cost of ~100 Mb. We use this to find the correct page of the disk, then +run the interpolation search in that page only. That approach allows us to find +the correct value with a single read in the data file.

Conclusion

This new backend is now used byt the Tezos nodes in production, and manages to +reduce the storage size from 250 Gb down to 25 Gb, with a monthly growth +rate of 2 Gb , achieving a tenfold reduction.

In the meantime, it provides and single writer, multiple readers access pattern +that enables bakers and clients to connect to the same storage while it is operated.

On the memory side, all our components are memory bounded, and the bound is +generally customizable, the largest source of memory usage being the log part +of the index. While it can be reduced to fit in 1 Gb of memory and run on +small VPS or Raspberry Pi, one can easily set a higher memory limit on a more +powerful machine, and achieve even better time performance.

Fuzzing OCamlFormat with AFL and Crowbar

tarides — 2020-08-03T00:00:00-00:00

AFL (and fuzzing in general) is often used +to find bugs in low-level code like parsers, but it also works very well to find +bugs in high level code, provided the right ingredients. We applied this +technique to feed random programs to OCamlFormat and found many formatting bugs.

OCamlFormat is a tool to format source code. To do so, it parses the source code +to an Abstract Syntax Tree (AST) and then applies formatting rules to the AST.

It can be tricky to correctly format the output. For example, say we want to +format (a+b)*c. The corresponding AST will look like Apply("*", Apply ("+", Var "a", Var "b"), Var "c"). A naive formatter would look like this:

let rec format = function
+  | Var s -> s
+  | Apply (op, e1, e2) ->
+      Printf.sprintf "%s %s %s" (format e1) op (format e2)

But this is not correct, as it will print (a+b)*c as a+b*c, which is a +different program. In this particular case, the common solution would be to +track the relative precedence of the expressions and to emit only necessary +parentheses.

OCamlFormat has similar cases. To make sure we do not change a program when +formatting it, there is an extra check at the end to parse the output and +compare the output AST with the input AST. This ensures that, in case of bugs, +OCamlFormat exits with an error rather than changing the meaning of the input +program.

When we consider the whole OCaml language, the rules are complex and it is +difficult to make sure that we are correctly handling all programs. There are +two main failure modes: either we put too many parentheses, and the program does +not look good, or we do not put enough, and the AST changes (and OCamlFormat +exits with an error). We need a way to make sure that the latter does not +happen. Tests work to some extent, but some edge cases happen only when a +certain combination of language features is used. Because of this combinatorial +explosion, it is impossible to get good coverage using tests only.

Fortunately there is a technique we can use to automatically explore the program +space: fuzzing. For a primer on using this technique on OCaml programs, one can +refer to this article.

To make this work we need two elements: a random program generator, and a +property to check. Here, we are interested in programs that are valid (in the +sense that they parse correctly) but do not format correctly. We can use the +OCamlFormat internals to do the following:

try to parse input: in case of a parse error, just reject this input as

invalid.

otherwise, with have a valid program. try to format it. If this happens with

no error at all, reject this input as well.

otherwise, it means that the AST changed, comments moved, or something

similar, in a valid program. This is what we are after.

Generating random programs is a bit more difficult. We can feed random strings +to AFL, but even with a corpus of existing valid code it will generate many +invalid programs. We are not interested in these for this project, we would +rather start from valid programs.

A good way to do that is to use Crowbar to directly generate AST values. Thanks +to ppx_deriving_crowbar and ppx_import +it is possible to generate random values for an external type like +Parsetree.structure (the contents of .ml files). Even more fortunately +somebody already did the work. Thanks, Mindy!

This approach works really well: it generates 5k-10k programs per second, which +is very good performance (AFL starts complaining below 100/s).

Quickly, AFL was able to find crashes related to attributes. These are "labels" +attached to various nodes of the AST. For example the expression (x || y) [@a] +(logical or between x and y, attach attribute a to the "or" expression) +would get formatted as x || y [@a] (attribute a is attached to the y +variable). Once again, there is a check in place in OCamlFormat to make sure +that it does not save the file in this case, but it would exit with an error.

After the fuzzer has run for a bit longer, it found crashes where comments would +jump around in expressions like f (*a*) (*bb*) x. Wait, what? We never told +the program generator how to generate comments. Inspecting the intermediate AST, +the part in the middle is actually an integer literal with value "(*a*) (*bb*)" (integer literals are represented as strings so that a third party +library could add literals for arbitrary precision numbers for +example).

AFL comes with a program called afl-tmin that is used to minimize a crash. It +will try to find a smaller example of a program that crashes OCamlFormat. It +works well even with Crowbar in between. For example it is able to turn (new aaaaaa & [0;0;0;0])[@aaaaaaaaaa] into (0&0)[@a] (neither AFL nor OCamlFormat +knows about types, so they can operate on nonsensical programs. Finding a +well-typed version of a crash is usually not very difficult, but it has to be +done manually).

In total, letting AFL run overnight on a single core (that is relatively short +in terms of fuzzing) caused 453 crashes. After minimization and deduplication, +this corresponded to about 30 unique issues.

Most of them are related to attributes that OCamlFormat did not try to include +in the output, or where it forgot to add parentheses. Fortunately, there are +safeguards in OCamlFormat: since it checks that the formatting preserves the AST +structure, it will exit with an error instead of outputting a different program.

Once again, fuzzing has proved itself as a powerful technique to find actual +bugs (including high-level ones). A possible approach for a next iteration is to +try to detect more problems during formatting, such as finding cases where lines +are longer than allowed. It is also possible to extend the random program +generator so that it tries to generate comments, and let OCamlFormat check that +they are all laid out correctly in the output. We look forward to employing +fuzzing more extensively for OCamlFormat development in future.

The future of Tezos on MirageOS

tarides — 2020-04-20T00:00:00-00:00

We are very glad to announce that Tarides has been awarded two new grants from +the Tezos Foundation.

Thanks to these new grants, Tarides will continue to work on the integration +between Tezos and MirageOS. We believe that the secure deployment of blockchains +is still a major challenge today, and that deploying Tezos as a unikernel will +have a big impact in term of safety and security. It will be a key +differentiator that will separate Tezos from other blockchains.

The Tezos codebase is written in OCaml and is currently using more than 100 +external packages, among which one third comes from the MirageOS project. +However, it still heavily depends on non-compatible Unix libraries. Making the +Tezos codebase fully compatible with MirageOS will help Tezos with: distribution +and packaging, portability, secure deployment and operational safety.

We’ll regularly publish development progress updates, so stay tuned!

Tarides wins the FIC 2020 startup award

tarides — 2019-12-11T00:00:00-00:00

We are very excited to announce that Tarides has won an +award +from the International Cybersecurity Forum (FIC 2020).

Organized every year in Lille (France), the International +Cybersecurity Forum has become the leading European event on +cybersecurity and digital trust. Its main goal is to foster reflection +and exchanges within the European cybersecurity ecosystem.

We are very happy to have won the "Coup de Coeur" Prize, which will +bring great visibility to our technological innovations. It is also an +opportunity for us to meet experts in the cybersecurity sector and to +consider additional use-cases for our work. We would like to thank the +CEIS for organising the event and the members of +the jury for commending MirageOS and +OSMOSE.

The next FIC will be held on the 28th, 29th and 30th of January +2020 in Lille. For more details about the FIC and to register, visit +their website.

+
+

+ + + + +

MirageOS talk at the Paris Open Source Summit

tarides — 2019-12-04T00:00:00-00:00

We are thrilled to have been selected by the Paris Open Source Summit +committee to talk about “Secure-by-design IoT applications using MirageOS”.

The Paris Open Source Summit is an annual event where you can connect to +open-source communities and learn from tech leaders, project committers and +CTOs about the latest technical solutions, innovative uses and societal +challenges of open digital technology.

Thomas Gazagnaire, Tarides CEO/CTO, will explain what makes MirageOS a good +framework to build IoT applications and how we can use embedded devices running +on ARMv8, ESP32 or RISC-V to run secure and end-to-end open-source +infrastructure services such as VPN proxies and email servers. He will also +highlight how this infrastructure will be used to form the basis of OSMOSE: a +secure, distributed and privacy-preserving platform to write user-centric IoT +applications.

MirageOS is a library operating system (using the MIT license) which enables the +construction of unikernels: specialized services where the runtime binary +contains only the necessary code for execution and no more. Unikernels have a +drastically smaller attack surface than service deployments in traditional +operating systems and could lead to 1000x less code for the full application +stack. Moreover, as MirageOS is written in a memory safe language (OCaml), a +full class of bugs related to memory corruption – representing 70% of the +released CVEs in classic operating systems written in C – +can no longer appear. These two properties combined (and more!) allow MirageOS +to build “secure-by-design” applications where everything – from the high-level +business logic to the low-level device drivers – has been designed to be as +secure as possible.

To learn more about the project, attend the Paris Open Source Summit! The talk +will take place during the 'Embedded & IOT' section +at 14:50 – 15:20 on December 10th, 2019.

+
+

+ + + + +

Introducing the GraphQL API for Irmin 2.0

tarides — 2019-11-27T00:00:00-00:00

With the release of Irmin 2.0.0, we are happy to announce a new package - irmin-graphql, which can be used to serve data from Irmin over HTTP. This blog post will give you some examples to help you get started, there is also a section in the irmin-tutorial with similar information. To avoid writing the same thing twice, this post will cover the basics of getting started, plus a few interesting ideas for queries.

Getting the irmin-graphql server running from the command-line is easy:

$ irmin graphql --root=/tmp/irmin

where /tmp/irmin is the actual path to your repository. This will start the server on localhost:8080, but it's possible to customize this using the --address and --port flags.

The new GraphQL API has been added to address some of the shortcomings that have been identified with the old HTTP API, as well as enable a number of new features and capabilities.

GraphQL

GraphQL is a query language for exposing data as a graph via an API, typically using HTTP as a transport. The centerpiece of a GraphQL API is the schema, which describes the graph in terms of types and relationships between these types. The schema is accessible by the consumer, and acts as a contract between the API and the consumer, by clearly defining all API operations and fully assigning types to all interactions.

Viewing Irmin data as a graph turns out to be a natural and useful model. Concepts such as branches and commits fit in nicely, and the stored application data is organized as a tree. Such highly hierarchical data can be challenging to interact with using REST, but is easy to represent and navigate with GraphQL.

+ + + +(image from Pro Git)

As a consumer of an API, one of the biggest initial challenges is understanding what operations are exposed and how to use them. Conversely, as a developer of an API, keeping documentation up-to-date is challenging and time consuming. Though no substitute for more free-form documentation, a GraphQL schema provides an excellent base line for understanding a GraphQL API that is guaranteed to be accurate and up-to-date. This issue is definitely true of the old Irmin HTTP API, which was hard to approach for newcomers due to lack of documentation.

Being able to inspect the schema of a GraphQL API enables powerful tooling. A great example of this is GraphiQL, which is a browser-based IDE for GraphQL queries. GraphiQL can serve both as an interactive API explorer and query designer with intelligent autocompletion, formatting and more.

+ + + + +

The combination of introspection and a strongly typed schema also allows creating smart clients using code generation. This is already a quite wide-spread idea with Apollo for iOS, Apollo for Android or graphql_ppx for OCaml/Reason. Though generic GraphQL client libraries will do a fine job interacting with the Irmin GraphQL API, these highlighted libraries will offer excellent ergonomics and type-safety out of the box.

One of the problems that GraphQL set out to solve is that of over- and underfetching. When designing REST API response payloads, there is always a tension between including too little data, which will require clients to make more network requests, and including too much data, which wastes resources for both client and server (serialization, network transfer, deserialization, etc).
+The existing low-level Irmin HTTP API is a perfect example of this. Fetching the contents of a particular file on the master branch requires at least 4 HTTP requests (fetch the branch, fetch the commit, fetch the tree, fetch the blob), i.e. massive underfetching. By comparison, this is something easily solved with a single request to the new GraphQL API. More generally, the GraphQL API allows you to fetch exactly the data you need in a single request without making one-off endpoints.

For the curious, here's the GraphQL query to fetch the contents of README.md from the branch master:

query {
+  master {
+    tree {
+      get(key: "README.md")
+    }
+  }
+}

The response will look something like this:

{
+  "data": {
+    "master": {
+      "tree": {
+        "get": "The contents of README.md"
+      }
+    }
+  }
+}

The GraphQL API is not limited to only reading data, you can also write data to your Irmin store. Here's a simple example that will set the key README.md to "foo", and return the hash of that commit:

mutation {
+  set(key: "README.md", value: "foo") {
+    hash
+  }
+}

By default, GraphQL allows you to do multiple operations in a single query, so you get bulk operations for free. Here's a more complex example that modifies two different branches, branch-a and branch-b, and then merges branch-b into branch-a all in a single query:

mutation {
+  branch_a: set(branch: "branch-a", key: "foo", value: "bar") {
+    hash
+  }
+
+  branch_b: set(branch: "branch-a", key: "baz", value: "qux") {
+    hash
+  }
+
+  merge_with_branch(branch: "branch-b", from: "branch-a") {
+    hash
+    tree {
+      list_contents_recursively {
+        key
+        value
+      }
+    }
+  }
+}

Here's what the response might look like:

{
+  "data": {
+    "branch_a": {
+      "hash": "0a1313ae9dfe1d4339aee946dd76b383e02949b6"
+    },
+    "branch_b": {
+      "hash": "28855c277671ccc180c81058a28d3254f17d2f7b"
+    },
+    "merge_with_branch": {
+      "hash": "7b17437a16a858816d2710a94ccaa1b9c3506d1f",
+      "tree": {
+        "list_contents_recursively": [
+          {
+            "key": "/foo",
+            "value": "bar"
+          },
+          {
+            "key": "/baz",
+            "value": "qux"
+          }
+        ]
+      }
+    }
+  }
+}

Overall, the new GraphQL API operates at a much higher level than the old HTTP API, and offers a number of complex operations that were tricky to accomplish before.

Customizable

With GraphQL, all request and response data is fully described by the schema. Because Irmin allows the user to have custom content types, this leaves the question of what type to assign to such values. By default, the GraphQL API will expose all values as strings, i.e. the serialized version of the data that your application stores. This works quite well when Irmin is used as a simple key-value store, but it can be very inconvenient scheme when storing more complex values. As an example, consider storing contacts (name, email, phone, tags, etc) in your Irmin store, where values have the following type:

(* Custom content type: a contact *)
+type contact = {
+  name : string;
+  email : string;
+  (* ... *)
+}

Fetching such a value will by default be returned to the client as the JSON encoded representation. Assume we're storing a contact under the key john-doe, which we fetch with the following query:

query {
+  master {
+    tree {
+      get(key: "john-doe")
+    }
+  }
+}

The response would then look something like this:

{
+  "master": {
+    "tree": {
+      "get": "{\"name\":\"John Doe\", \"email\": \"john.doe@gmail.com/", ...}"
+    }
+  }
+}

The client will have to parse this JSON string and cannot choose to only fetch parts of the value (say, only the email). Optimally we would want the client to get a structured response such as the following:

{
+  "master": {
+    "tree": {
+      "get": {
+        "name": "John Doe",
+        "email": "john.doe@gmail.com",
+        ...
+      }
+    }
+  }
+}

To achieve this, the new GraphQL API allows providing an "output type" and an "input type" for most of the configurable types in your store (contents, key, metadata, hash, branch). The output type specifies how data is presented to the client, while the input type controls how data can be provided by the client. Let's take a closer look at specifying a custom output type.

Essentially you have to construct a value of type (unit, 'a option) Graphql_lwt.Schema.typ (from the graphql-lwt package), assuming your content type is 'a. We could construct a GraphQL object type for our example content type contact as follows:

(* (unit, contact option) Graphql_lwt.Schema.typ *)
+let contact_schema_typ = Graphql_lwt.Schema.(obj "Contact"
+  ~fields:(fun _ -> [
+    field "name"
+      ~typ:(non_null string)
+      ~args:[]
+      ~resolve:(fun _ contact ->
+        contact.name
+      )
+    ;
+    (* ... more fields *)
+  ])
+)

To use the custom type, you need to instantiate the functor Irmin_unix.Graphql.Server.Make_ext (assuming you're deploying to a Unix target) with an Irmin store (type Irmin.S) and a custom types module (type Irmin_graphql.Server.CUSTOM_TYPES). This requires a bit of plumbing:

(* Instantiate the Irmin functor somehow *)
+module S : Irmin.S with type contents = contact =
+  (* ... *)
+
+(* Custom GraphQL presentation module *)
+module Custom_types = struct
+  (* Construct default GraphQL types *)
+  module Defaults = Irmin_graphql.Server.Default_types (S)
+
+  (* Use the default types for most things *)
+  module Key = Defaults.Key
+  module Metadata = Defaults.Metadata
+  module Hash = Defaults.Hash
+  module Branch = Defaults.Branch
+
+  (* Use custom output type for contents *)
+  module Contents = struct
+    include Defaults.Contents
+    let schema_typ = contact_schema_typ
+  end
+end
+
+module Remote = struct
+  let remote = Some s.remote
+end
+
+module GQL = Irmin_unix.Graphql.Server.Make_ext (S) (Remote) (Custom_types)

With this in hand, we can now query specifically for the email of john-doe:

query {
+  master {
+    tree {
+      get(key: "john-doe") {
+        email
+      }
+    }
+  }
+}

... and get a nicely structured JSON response back:

{
+  "master": {
+    "tree": {
+      "get": {
+        "email": "john.doe@gmail.com"
+      }
+    }
+  }
+}

The custom types is very powerful and opens up for transforming or enriching the data at query time, e.g. geocoding the address of a contact, or checking an on-line status.

Watches

A core feature of Irmin is the ability to watch for changes to the underlying data store in real-time. irmin-graphql takes advantage of GraphQL subscriptions to expose Irmin watches. Subscriptions are a relative recent addition to the GraphQL spec (June 2018), which allows clients to subscribe to changes. These changes are pushed to the client over a suitable transport mechanism, e.g. websockets, Server-Sent Events, or a chunked HTTP response, as a regular GraphQL response.

As an example, the following query watches for all changes and returns the new hash:

subscription {
+  watch {
+    commit {
+      hash
+    }
+  }
+}

For every change, a message like the following will be sent:

{
+  "watch": {
+    "commit": {
+      "hash": "c01a59bacc16d89e9cdd344a969f494bb2698d8f"
+    }
+  }
+}

Under the hood, subscriptions in irmin-graphql are implemented using Irmin watches, but this is opaque to the client -- this will work with any GraphQL spec compliant client!

Here's a video, which hows how the GraphQL response changes live as the Irmin store is being manipulated:

Note that the current implementation only supports websockets with more transport options coming soon.

Wrap-up

Irmin 2.0 ships with a powerful new GraphQL API, that makes it much easier to interact with Irmin over the network. This makes Irmin available for many more languages and contexts, not just applications using OCaml (or Javascript). The new API operates at a much high level than the old API, and offers advanced features such as "bring your own GraphQL types", and watching for changes via GraphQL subscriptions.

We're looking forward to seeing what you'll build with it!

Irmin v2

tarides — 2019-11-21T00:00:00-00:00

We are pleased to announce Irmin +2.0.0, a major release of the +Git-like distributed branching and storage substrate that underpins +MirageOS. We began the release process for all the +components that make up Irmin back in May +2019, and there +have been close to 1000 commits since Irmin 1.4.0 released back in June 2018. To +celebrate this milestone, we have a new logo and opened a dedicated website: +irmin.org.

Our focus this year has been on ensuring the production success of our +early adopters -- such as the +Tezos blockchain +and the Datakit 9P +stack -- as well as spawning new research projects into the practical +application of distributed and mergeable data stores. We are also +very pleased to welcome several new maintainers into the Mirage +project for their contributions to Irmin, namely +Ioana Cristescu, +Craig Ferguson, +Andreas Garnaes, +Clément Pascutto and +Zach Shipko.

New Major Features

New CLI

While Irmin is normally used as a library, it is obviously useful to +be able to interact with a data store from a shell. The irmin-unix +opam package now provides an irmin binary that is configured via a +Yaml file and can perform queries and mutations against a Git store.

$ echo "root: ." > irmin.yml
+$ irmin init
+$ irmin set foo/bar "testing 123"
+$ irmin get foo/bar

Try irmin --help to see all the commands and options available.

Tezos and irmin-pack

Another big user of Irmin is the Tezos blockchain, +and we have been optimising the persistent space usage of Irmin as their +network grows. Because Tezos doesn’t require full Git format support, +we created a hybrid backend that grabs the best bits of Git (e.g. the +packfile mechanism) and engineered a domain-specific backend tailored +for Tezos usage. Crucially, because of the way Irmin is split into +clean libraries and OCaml modules, we only had to modify a small part +of the codebase and could also reuse elements of our +OCaml-git codebase as well.

The irmin-pack backend is available +for use in the CLI and provides a +significant improvement in disk usage. There is a corresponding Tezos merge +request using the Irmin +2.0 code that has been integrated downstream and will become available via +their release process in due course.

As part of this development process, we also released an efficient multi-level +index implementation (imaginatively dubbed +index in opam). Our implementation takes an +arbitrary IO implementation and user-supplied content types and supplies a +standard key-value interface for persistent storage. Index provides instance +sharing by default, so each OCaml runtime shares a common singleton instance.

Irmin-GraphQL and “browser Irmin”

Another new area of huge interest to us is +GraphQL in order to provide frontends with a rich +query language for Irmin-hosted applications. Irmin 2.0 includes a +built-in GraphQL server so you can manipulate your Git repo via +GraphQL.

If you are interested in (for example) compiling elements of Irmin to +JavaScript or wasm, for usage in frontends, then the Irmin 2.0 release +makes it significantly easier to support this architecture. We’ve +already seen some exploratory efforts report issues +when doing this, and we’ve had it working ourselves in Irmin 1.0 Cuekeeper +so we are excited by the potential power of applications built using +this model. If you have ideas/questions, please get in touch on the +issue tracker with your +usecase.

Wodan

Irmin’s storage layer is also well abstracted, so backends other than +a Unix filesystem or Git are supported. Irmin can run in highly +diverse and OS-free environments, and so we began engineering the +Wodan filesystem as a +domain-specific filesystem designed for MirageOS, Irmin and modern +flash drives. See the OCaml Workshop 2017 abstract on +it for more design +rationale.

As part of the Irmin 2.0 release, Wodan is also being prepared for a +release, and you can find Irmin 2.0 +support +in the source. If you’d like a standalone block-device based +persistence environment for Irmin, please try this out. This is the +preferred backend for using Irmin storage in a unikernel.

### Versioned CalDAV

An application pulling all these pieces together is being developed +by our friends at Robur: an Irmin-based +CalDAV calendaring server +that even hosts its DNS server using a versioned Irmin store. We'll +blog more about this as the components get released and stabilised, but +the unikernel enthusiasts among you may want to browse the +Robur unikernels future branch +to see how they are deploying them today.

A huge thank you to all our commercial customers, end users and open-source +developers who have contributed their time, expertise and +financial support to help us achieve our goal of delivering a modern +storage stack in the spirit of Git. Our next steps for Irmin are to +continue to increase the performance and optimise the storage, +and to build more end-to-end applications using the application core +on top of MirageOS.

Mr. MIME - Parse and generate emails

tarides — 2019-09-25T00:00:00-00:00

We're glad to announce the first release of mrmime, a parser and a +generator of emails. This library provides an OCaml way to analyze and craft +an email. The eventual goal is to build an entire unikernel-compatible stack +for email (such as SMTP or IMAP).

In this article, we will show what is currently possible with mrmime and +present a few of the useful libraries that we developed along the way.

An email parser

Some years ago, Romain gave a talk about what an email really is. +Behind the human-comprehensible format (or rich-document as we said a +long time ago), there are several details of emails which complicate the process of +analyzing them (and can be prone to security lapses). These details are mostly described +by three RFCs:

Even though they are cross-compatible, providing full legacy email parsing is an +archaeological exercise: each RFC retains support for the older design decisions +(which were not recognized as bad or ugly in 1970 when they were first standardized).

The latest email-related RFC (RFC5322) tried to fix the issue and provide a better +formal specification of the email format – but of course, it comes with plenty of +obsolete rules which need to be implemented. In the standard, you find +both the current grammar rule and its obsolete equivalent.

An extended email parser

Even if the email format can defined by "only" 3 RFCs, you will +miss email internationalization (RFC6532), the MIME format +(RFC2045, RFC2046, RFC2047, +RFC2049), or certain details needed to be interoperable with SMTP +(RFC5321). There are still more RFCs which add extra features +to the email format such as S/MIME or the Content-Disposition field.

Given this complexity, we took the most general RFCs and tried to provide an easy way to deal +with them. The main difficulty is the multipart parser, which deals with email +attachments (anyone who has tried to make an HTTP 1.1 parser knows about this).

A realistic email parser

Respecting the rules described by RFCs is not enough to be able to analyze any +email from the real world: existing email generators can, and do, produce +non-compliant email. We stress-tested mrmime by feeding it a batch of 2 +billion emails taken from the wild, to see if it could parse everything (even if +it does not produce the expected result). Whenever we noticed a recurring +formatting mistake, we updated the details of the ABNF to enable +mrmime to parse it anyway.

A parser usable by others

One demonstration of the usability of mrmime is ocaml-dkim, which wants to +extract a specific field from your mail and then verify that the hash and signature +are as expected.

ocaml-dkim is used by the latest implementation of ocaml-dns to request +public keys in order to verify email.

The most important question about ocaml-dkim is: is it able to +verify your email in one pass? Indeed, currently some implementations of DKIM +need 2 passes to verify your email (one to extract the DKIM signature, the other +to digest some fields and bodies). We focused on verifying in a single pass in +order to provide a unikernel SMTP relay with no need to store your email between +verification passes.

An email generator

OCaml is a good language for making little DSLs for specialized use-cases. In this +case, we took advantage of OCaml to allow the user to easily craft an email from +nothing.

The idea is to build an OCaml value describing the desired email header, and +then let the Mr. MIME generator transform this into a stream of characters that +can be consumed by, for example, an SMTP implementation. The description step +is quite simple:

#require "mrmime" ;;
+#require "ptime.clock.os" ;;
+
+open Mrmime
+
+let romain_calascibetta =
+  let open Mailbox in
+  Local.[ w "romain"; w "calascibetta" ] @ Domain.(domain, [ a "gmail"; a "com" ])
+
+let john_doe =
+  let open Mailbox in
+  Local.[ w "john" ] @ Domain.(domain, [ a "doe"; a "org" ])
+  |> with_name Phrase.(v [ w "John"; w "D." ])
+
+let now () =
+  let open Date in
+  of_ptime ~zone:Zone.GMT (Ptime_clock.now ())
+
+let subject =
+  Unstructured.[ v "A"; sp 1; v "Simple"; sp 1; v "Mail" ]
+
+let header =
+  let open Header in
+  Field.(Subject $ subject)
+  & Field.(Sender $ romain_calascibetta)
+  & Field.(To $ Address.[ mailbox john_doe ])
+  & Field.(Date $ now ())
+  & empty
+
+let stream = Header.to_stream header
+
+let () =
+  let rec go () =
+    match stream () with
+    | Some buf -> print_string buf; go ()
+    | None -> ()
+  in
+  go ()

This code produces the following header:

Date: 2 Aug 2019 14:10:10 GMT
+To: John "D." <john@doe.org>
+Sender: romain.calascibetta@gmail.com
+Subject: A Simple Mail

78-character rule

One aspect about email and SMTP is about some historical rules of how to +generate them. One of them is about the limitation of bytes per line. Indeed, a +generator of mail should emit at most 80 bytes per line - and, of course, it +should emits entirely the email line per line.

So mrmime has his own encoder which tries to wrap your mail into this limit. +It was mostly inspired by Faraday and Format powered with +GADT to easily describe how to encode/generate parts of an email.

A multipart email generator

Of course, the main point about email is to be able to generate a multipart +email - just to be able to send file attachments. And, of course, a deep work +was done about that to make parts, compose them into specific Content-Type +fields and merge them into one email.

Eventually, you can easily make a stream from it, which respects rules (78 bytes +per line, stream line per line) and use it directly into an SMTP implementation.

This is what we did with the project facteur. It's a little +command-line tool to send with file attachement mails in pure OCaml - but it +works only on an UNIX operating system for instance.

Behind the forest

Even if you are able to parse and generate an email, more work is needed to get the expected results.

Indeed, email is a exchange unit between people and the biggest deal on that is +to find a common way to ensure a understable communication each others. About +that, encoding is probably the most important piece and when a French person wants +to communicate with a latin1 encoding, an American person can still use ASCII.

Rosetta

So about this problem, the choice was made to unify any contents to UTF-8 as the +most general encoding of the world. So, we did some libraries which map an encoding flow +to Unicode code-point, and we use uutf (thanks to dbuenzli) to normalize it to UTF-8.

The main goal is to avoid a headache to the user about that and even if +contents of the mail is encoded with latin1 we ensure to translate it +correctly (and according RFCs) to UTF-8.

This project is rosetta and it comes with:

uuuu for ISO-8859 encoding
coin for KOI8-{R,U} encoding
yuscii for UTF-7 encoding

Pecu and Base64

Then, bodies can be encoded in some ways, 2 precisely (if we took the main +standard):

A base64 encoding, used to store your file
A quoted-printable encoding

So, about the base64 package, it comes with a sub-package base64.rfc2045 +which respects the special case to encode a body according RFC2045 and SMTP +limitation.

Then, pecu was made to encode and decode quoted-printable contents. It was +tested and fuzzed of course like any others MirageOS's libraries.

These libraries are needed for an other historical reason which is: bytes used +to store mail should use only 7 bits instead of 8 bits. This is the purpose of +the base64 and the quoted-printable encoding which uses only 127 possibilities +of a byte. Again, this limitation comes with SMTP protocol.

Conclusion

mrmime is tackling the difficult task to parse and generate emails according to 50 years of usability, several RFCs and legacy rules. +So, it +still is an experimental project. We reach the first version of it because we +are currently able to parse many mails and then generate them correctly.

Of course, a bug (a malformed mail, a server which does not respect standards +or a bad use of our API) can appear easily where we did not test everything. But +we have the feeling it was the time to release it and let people to use +it.

The best feedback about mrmime and the best improvement is yours. So don't be +afraid to use it and start to hack your emails with it.

Decompress: Experiences with OCaml optimization

tarides — 2019-09-13T00:00:00-00:00

In our first article we mostly discussed +the API design of decompress and did not talk too much about the issue of +optimizing performance. In this second article, we will relate our experiences +of optimizing decompress.

As you might suspect, decompress needs to be optimized a lot. It was used by +several projects as an underlying layer of some formats (like Git), so it can be +a real bottleneck in those projects. Of course, we start with a footgun by using +a garbage-collected language; comparing the performance of decompress with a C +implementation (like zlib or miniz) is obviously not very fair.

However, using something like decompress instead of C implementations can be +very interesting for many purposes, especially when thinking about unikernels. +As we said in the previous article, we can take the advantage of the runtime +and the type-system to provide something safer (of course, it's not really +true since zlib has received several security audits).

The main idea in this article is not to give snippets to copy/paste into your +codebase but to explain some behaviors of the compiler / runtime and hopefully +give you some ideas about how to optimize your own code. We'll discuss the +following optimizations:

specialization
inlining
untagged integers
exceptions
unrolling
hot-loop
caml_modify
representation sizes

Cautionary advice

Before we begin discussing optimization, keep this rule in mind:

+
Only perform optimization at the end of the development process.
+

An optimization pass +can change your code significantly, so you need to keep a state of your project +that can be trusted. This state will provide a comparison point for both +benchmarks and behaviors. In other words, your stable implementation will be the +oracle for your benchmarks. If you start with nothing, you'll achieve +arbitrarily-good performance at the cost of arbitrary behavior!

We optimized decompress because we are using it in bigger projects for a long +time (2 years). So we have an oracle (even if zlib can act as an oracle in +this special case).

Specialization

One of the biggest specializations in decompress is regarding the min +function. If you don't know, in OCaml min is polymorphic; you can compare +anything. So you probably have some concerns about how min is implemented?

You are right to be concerned: if you examine the details, min calls the C +function do_compare_val, which traverses your structure and does a comparison +according the run-time representation of your structure. Of course, for integers, it +should be only a cmpq assembly instruction. However, some simple code like:

let x = min 0 1

will produce this CMM and assembly code:

(let x/1002 (app{main.ml:1,8-15} "camlStdlib__min_1028" 1 3 val)
+   ...)

.L101:
+        movq    $3, %rbx
+        movq    $1, %rax
+        call    camlStdlib__min_1028@PLT

Note that beta-reduction, inlining and +specialization were not done in this code. OCaml does not optimize your code +very much – the good point is predictability of the produced assembly output.

If you help the compiler a little bit with:

external ( <= ) : int -> int -> bool = "%lessequal"
+let min a b = if a <= b then a else b [@@inline]
+
+let x = min 0 1

We have:

(function{main.ml:2,8-43} camlMain__min_1003 (a/1004: val b/1005: val)
+ (if (<= a/1004 b/1005) a/1004 b/1005))
+
+(function camlMain__entry ()
+ (let x/1006 1 (store val(root-init) (+a "camlMain" 8) 1)) 1a)

.L101:
+        cmpq    %rbx, %rax
+        jg      .L100
+        ret

So we have all optimizations, in this produced code, x was evaluated as 0 +(let x/... (store ... 1)) (beta-reduction and inlining) and min was +specialized to accept only integers – so we are able to emit cmpq.

Results

With specialization, we won 10 Mb/s on decompression, where min is used +in several places. We completely avoid an indirection and a call to the slow +do_compare_val function.

This kind of specialization is already done by flambda, however, we +currently use OCaml 4.07.1. So we decided to this kind of optimization by +ourselves.

Inlining

In the first example, we showed code with the [@@inline] keyword which is +useful to force the compiler to inline a little function. We will go outside the +OCaml world and study C code (gcc 5.4.0) to really understand +inlining.

In fact, inlining is not necessarily the best optimization. Consider the +following (nonsensical) C program:

#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <time.h>
+#include <stdlib.h>
+
+#ifdef HIDE_ALIGNEMENT
+__attribute__((noinline, noclone))
+#endif
+void *
+hide(void * p) { return p; }
+
+int main(int ac, const char *av[])
+{
+  char *s = calloc(1 << 20, 1);
+  s = hide(s);
+
+  memset(s, 'B', 100000);
+
+  clock_t start = clock();
+
+  for (int i = 0; i < 1280000; ++i)
+    s[strlen(s)] = 'A';
+
+  clock_t end = clock();
+
+  printf("%lld\n", (long long) (end-start));
+
+  return 0;
+}

We will compile this code with -O2 (the second level of optimization in C), +once with -DHIDE_ALIGNEMENT and once without. The assembly emitted differs:

.L3:
+	movq	%rbp, %rdi
+	call	strlen
+	subl	$1, %ebx
+	movb	$65, 0(%rbp,%rax)
+	jne	.L3

.L3:
+	movl	(%rdx), %ecx
+	addq	$4, %rdx
+	leal	-16843009(%rcx), %eax
+	notl	%ecx
+	andl	%ecx, %eax
+	andl	$-2139062144, %eax
+	je	.L3

In the first output (with -DHIDE_ALIGNEMENT), the optimization pass +decides to disable inlining of strlen; in the second output (without +-DHIDEAlIGNEMENT), it decides to inline strlen (and do some other clever +optimizations). The reason behind this complex behavior from the compiler is +clearly described here.

But what we want to say is that inlining is not an automatic optimization; +it might act as a pessimization. This is the goal of flambda: do the right +optimization under the right context. If you are really curious about what gcc +does and why, even if it's very interesting, the reverse engineering of the +optimization process and which information is relevant about the choice to +optimize or not is deep, long and surely too complicated.

A non-spontaneous optimization is to annotate some parts of your code with +[@@inline never] – so, explicitly say to the compiler to not inline the +function. This constraint is to help the compiler to generate a smaller code +which will have more chance to fit under the processor cache.

For all of these reasons, [@@inline] should be used sparingly and an oracle to +compare performances if you inline or not this or this function is necessary to +avoid a pessimization.

In `decompress`

Inlining in decompress was done on small functions which need to allocate +to return a value. If we inline them, we can take the opportunity to store +returned value in registers (of course, it depends how many registers are free).

As we said, the goal of the inflator is to translate a bit sequence to a byte. +The largest bit sequence possible according to RFC 1951 has length 15. So, when +we process an inputs flow, we eat it 15 bits per 15 bits. For each packet, we +want to recognize an existing associated bit sequence and then, binded values +will be the real length of the bit sequence and the byte:

val find : bits:int -> { len: int; byte: int; }

So for each call to this function, we need to allocate a record/tuple. It's +why we choose to inline this function. min was inlined too and some other +small functions. But as we said, the situation is complex; where we think that +inlining can help us, it's not systematically true.

NOTE: we can recognize bits sequence with, at most, 15 bits because a +Huffman coding is prefix-free.

Untagged integers

When reading assembly, the integer 0 is written as $1. +It's because of the GC bit needed to differentiate a pointer +and an unboxed integer. This is why, in OCaml, we talk about a 31-bits integer +or a 63-bits integer (depending on your architecture).

We will not try to start a debate about this arbitrary choice on the +representation of an integer in OCaml. However, we can talk about some +operations which can have an impact on performances.

The biggest example is about the mod operation. Between OCaml and C, % or +mod should be the same:

let f a b = a mod b

The output assembly is:

.L105:
+        movq    %rdi, %rcx
+        sarq    $1, %rcx     // b >> 1
+        movq    (%rsp), %rax
+        sarq    $1, %rax     // a >> 1
+        testq   %rcx, %rcx   // b != 0
+        je      .L107
+        cqto
+        idivq   %rcx         // a % b
+        jmp     .L106
+.L107:
+        movq    caml_backtrace_pos@GOTPCREL(%rip), %rax
+        xorq    %rbx, %rbx
+        movl    %ebx, (%rax)
+        movq    caml_exn_Division_by_zero@GOTPCREL(%rip), %rax
+        call    caml_raise_exn@PLT
+.L106:
+        salq    $1, %rdx     // x << 1
+        incq    %rdx         // x + 1
+        movq    %rbx, %rax

where idiomatically the same C code produce:

.L2:
+        movl    -12(%rbp), %eax
+        cltd
+        idivl   -8(%rbp)
+        movl    %edx, -4(%rbp)

Of course, we can notice firstly the exception in OCaml (Divided_by_zero) - +which is pretty good because it protects us against an interrupt from assembly +(and keep the trace). Then, we need to untag a and b with sarq assembly +operation. We do, as the C code, idiv and then we must retag returned value +x with salq and incq.

So in some parts, it should be more interesting to use Nativeint. However, by +default, a nativeint is boxed. boxed means that the value is allocated in +the OCaml heap alongside a header.

Of course, this is not what we want so, if our nativeint ref (to have +side-effect, like x) stay inside a function and then, you return the real +value with the deref ! operator, OCaml, by a good planet alignment, can +directly use registers and real integers. So it should be possible to avoid +these needed conversions.

Readability versus performance

We use this optimization only in few parts of the code. In fact, switch +between int and nativeint is little bit noisy:

hold := Nativeint.logor !hold Nativeint.(shift_left (of_int (unsafe_get_uint8 d.i !i_pos)) !bits)

In the end, we only gained 0.5Mb/s of inflation rate, so it's not worthwhile +to do systematically this optimization. Especially that the gain is not very +big. But this case show a more troubling problem: loss of readability.

In fact, we can optimize more and more a code (OCaml or C) but we lost, step by +step, readability. You should be afraid by the implementation of strlen for +example. In the end, the loss of readability makes it harder to understand the purpose +of the code, leading to errors whenever some other person (or you in 10 years time) +tries to make a change.

And we think that this kind of optimization is not the way of OCaml in general +where we prefer to produce an understandable and abstracted code than a cryptic +and super fast one.

Again, flambda wants to fix this problem and let the compiler to do this +optimization. The goal is to be able to write a fast code without any pain.

Exceptions

If you remember our article about the release of base64, we talked a +bit about exceptions and used them as a jump. In fact, it's pretty +common for an OCaml developer to break the control-flow with an exception. +Behind this common design/optimization, it's about calling convention.

Indeed, choose the jump word to describe OCaml exception is not the best where +we don't use setjmp/longjmp.

In the details, when you start a code with a try .. with, OCaml saves a trap +in the stack which contains information about the with, the catcher. Then, +when you raise, you jump directly to this trap and can just discard several +stack frames (and, by this way, you did not check each return codes).

In several places and mostly in the hot-loop, we use this pattern. However, +it completely breaks the control flow and can be error-prone.

To limit errors and because this pattern is usual, we prefer to use a local +exception which will be used only inside the function. By this way, we enforce +the fact that exception should not (and can not) be caught by something else +than inside the function.

    let exception Break in
+
+    ( try while !max >= 1 do
+          if bl_count.(!max) != 0 then raise_notrace Break
+        ; decr max done with Break -> () ) ;

This code above produce this assembly code:

.L105:
+        pushq   %r14
+        movq    %rsp, %r14
+.L103:
+        cmpq    $3, %rdi              // while !max >= 1
+        jl      .L102
+        movq    -4(%rbx,%rdi,4), %rsi // bl_count,(!max)
+        cmpq    $1, %rsi              // bl_count.(!max) != 0
+        je      .L104
+        movq    %r14, %rsp
+        popq    %r14
+        ret                           // raise_notrace Break
+.L104:
+        addq    $-2, %rdi             // decr max
+        movq    %rdi, 16(%rsp)
+        jmp     .L103

Where the ret is the raise_notrace Break. A raise_notrace is needed, +otherwise, you will see:

        movq    caml_backtrace_pos@GOTPCREL(%rip), %rbx
+        xorq    %rdi, %rdi
+        movl    %edi, (%rbx)
+        call    caml_raise_exn@PLT

Instead the ret assembly code. Indeed, in this case, we need to store where we +raised the exception.

Unrolling

When we showed the optimization done by gcc when the string is aligned, gcc +did another optimization. Instead of setting the string byte per byte, it decides to +update it 4 bytes per 4 bytes.

This kind of this optimization is an unroll and we did it in decompress. +Indeed, when we reach the copy opcode emitted by the lz77 +compressor, we want to blit length byte(s) from a source to the outputs +flow. It can appear that this memcpy can be optimized to copy 4 bytes per 4 +bytes – 4 bytes is generally a good idea where it's the size of an int32 and +should fit under any architectures.

let blit src src_off dst dst_off =
+  if dst_off – src_off < 4
+  then slow_blit src src_off dst dst_off
+  else
+    let len0 = len land 3 in
+    let len1 = len asr 2 in
+
+    for i = 0 to len1 – 1
+    do
+      let i = i * 4 in
+      let v = unsafe_get_uint32 src (src_off + i) in
+      unsafe_set_uint32 dst (dst_off + i) v ;
+    done ;
+
+    for i = 0 to len0 – 1
+    do
+      let i = len1 * 4 + i in
+      let v = unsafe_get_uint8 src (src_off + i) in
+      unsafe_set_uint8 dst (dst_off + i) v ;
+    done

In this code, at the beginning, we copy 4 bytes per 4 bytes and if len is not +a multiple of 4, we start the trailing loop to copy byte per byte then. In +this context, OCaml can unbox int32 and use registers. So this function does +not deal with the heap, and by this way, with the garbage collector.

Results

In the end, we gained an extra 10Mb/s of inflation rate. The blit function is the +most important function when it comes to inflating the window to an output flow. +As the specialization on the min function, this is one of the biggest optimization on +decompress.

hot-loop

A common design about decompression (but we can find it on hash implementation +too), is the hot-loop. An hot-loop is mainly a loop on the most common +operation in your process. In the context of decompress, the hot-loop is +about a repeated translation from bits-sequence to byte(s) from the inputs flow +to the outputs flow and the window.

The main idea behind the hot-loop is to initialize all information needed for +the translation before to start the hot-loop. Then, it's mostly an imperative +loop with a pattern-matching which corresponds to the current state of the +global computation.

In OCaml, we can take this opportunity to use int ref (or nativeint ref), and then, they will be translated into registers (which is the fastest +area to store something).

Another deal inside the hot-loop is to avoid any allocation – and it's why we +talk about int or nativeint. Indeed, a more complex structure like an option +will add a blocker to the garbage collection (a call to caml_call_gc).

Of course, this kind of design is completely wrong if we think in a functional +way. However, this is the (biggest?) advantage of OCaml: hide this ugly/hacky +part inside a functional interface.

In the API, we talked about a state which represents the inflation (or the +deflation). At the beginning, the goal is to store into some references +essentials values like the position into the inputs flow, bits available, +dictionary, etc. Then, we launch the hot-loop and only at the end, we update the state.

So we keep the optimal design about inflation and the functional way outside +the hot-loop.

caml_modify

One issue that we need to consider is the call to caml_modify. In +fact, for a complex data-structure like an int array or a int option (so, +other than an integer or a boolean or an immediate value), values can move to the +major heap.

In this context, caml_modify is used to assign a new value into your mutable +block. It is a bit slower than a simple assignment but needed to +ensure pointer correspondence between minor heap and major heap.

With this OCaml code for example:

type t = { mutable v : int option }
+
+let f t v = t.v <- v

We produce this assembly:

camlExample__f_1004:
+        subq    $8, %rsp
+        movq    %rax, %rdi
+        movq    %rbx, %rsi
+        call    caml_modify@PLT
+        movq    $1, %rax
+        addq    $8, %rsp
+        ret

Where we see the call to caml_modify which will be take care about the +assignment of v into t.v. This call is needed mostly because the type of t.v is not an immediate value like an integer. So, for many values in the +inflator and the deflator, we mostly use integers.

Of course, at some points, we use int array and set them at some specific +points of the inflator – where we inflated the dictionary. However, the impact +of caml_modify is not very clear where it is commonly pretty fast.

Sometimes, however, it can be a real bottleneck in your computation and +this depends on how long your values live in the heap. A little program (which is +not very reproducible) can show that:

let t = Array.init (int_of_string Sys.argv.(1)) (fun _ -> Random.int 256)
+
+let pr fmt = Format.printf fmt
+
+type t0 = { mutable v : int option }
+type t1 = { v : int option }
+
+let f0 (t0 : t0) =
+  for i = 0 to Array.length t – 1
+  do let v = match t0.v, t.(i) with
+             | Some _ as v, _ -> v
+             | None, 5 -> Some i
+             | None, _ -> None in
+     t0.v <- v
+  done; t0
+
+let f1 (t1 : t1) =
+  let t1 = ref t1 in
+  for i = 0 to Array.length t – 1
+  do let v = match !t1.v, t.(i) with
+             | Some _ as v, _ -> v
+             | None, 5 -> Some i
+             | None, _ -> None in
+     t1 := { v }
+  done; !t1
+
+let () =
+  let t0 : t0 = { v= None } in
+  let t1 : t1 = { v= None } in
+  let time0 = Unix.gettimeofday () in
+  ignore (f0 t0) ;
+  let time1 = Unix.gettimeofday () in
+  ignore (f1 t1) ;
+  let time2 = Unix.gettimeofday () in
+
+  pr "f0: %f ns\n%!" (time1 -. time0) ;
+  pr "f1: %f ns\n%!" (time2 -. time1) ;
+
+  ()

In our bare-metal server, if you launch the program with 1000, the f0 +computation, even if it has caml_modify will be the fastest. However, if you +launch the program with 1000000000, f1 will be the fastest.

$ ./a.out 1000
+f0: 0.000006 ns
+f1: 0.000015 ns
+$ ./a.out 1000000000
+f0: 7.931782 ns
+f1: 5.719370 ns

About `decompress`

At the beginning, our choice was made to have, as @dbuenzli does, mutable +structure to represent state. Then, @yallop did a big patch to update it to an +immutable state and we won 9Mb/s on inflation.

However, the new version is more focused on the hot-loop and it is 3 +times faster than before.

As we said, the deal about caml_modify is not clear and depends a lot about +how long your data lives in the heap and how many times you want to update it. +If we localize caml_modify only on few places, it should be fine. But it still +is one of the most complex question about (macro?) optimization.

Smaller representation

We've discussed the impact that integer types can have on the use of immediate +values. More generally, the choice of type to represent your values can have +significant performance implications.

For example, a dictionary which associates a bits-sequence (an integer) to the +length of it AND the byte, it can be represented by a: (int * int) array, or +more idiomatically { len: int; byte: int; } array (which is structurally the +same).

However, that means an allocation for each bytes to represent every bytes. +Extraction of it will need an allocation if find : bits:int -> { len: int; byte: int; } is not inlined as we said. And about memory, the array can be +really heavy in your heap.

At this point, we used spacetime to show how many blocks we allocated for a +common inflation and we saw that we allocate a lot. The choice was made to use +a smaller representation. Where len can not be upper than 15 according RFC 1951 +and when byte can represent only 256 possibilities (and should fit under one +byte), we can decide to merge them into one integer (which can have, at least, +31 bits).

let static_literal_tree = [| (8, 12); (8, 140); (8, 76); ... |]
+let static_literal_tree = Array.map (fun (len, byte) -> (len lsl 8) lor byte) static_literal_tree

In the code above, we just translate the static dictionary (for a STATIC DEFLATE +block) to a smaller representation where len will be the left part of the +integer and byte will be the right part. Of course, it's depends on what you +want to store.

Another point is readability. cstruct-ppx and +bitstring can help you but decompress +wants to depend only on OCaml.

Conclusion

We conclude with some closing advice about optimising your OCaml programs:

+
Optimization is specific to your task. The points highlighted in this +article may not fit your particular problem, but they are intended to give you +ideas. Our optimizations were only possible because we completely assimilated +the ideas of zlib and had a clear vision of what we really needed to +optimize (like blit). +

+As your first project, this article can not help you a lot to optimize your +code where it's mostly about micro-optimization under a specific context +(hot-loop). But it helps you to understand what is really done by the +compiler – which is still really interesting.
+
+
Optimise only with respect to an oracle. All optimizations were done +because we did a comparison point between the old implementation of +decompress and zlib as oracles. Optimizations can change the semantics of your +code and you should systematically take care at any step about expected +behaviors. So it's a long run.
+
+
Use the predictability of the OCaml compiler to your advantage. For sure, +the compiler does not optimize a lot your code – but it sill produce realistic +programs if we think about performance. For many cases, you don't need to +optimize your OCaml code. And the good point is about expected behavior. +

+The mind-link between the OCaml and the assembly exists (much more than the C +and the assembly sometimes where we let the C compiler to optimize the code). +The cool fact is to keep a mental-model about what is going on on your code +easily without to be afraid by what the compiler can produce. And, in some +critical parts like eqaf, it's really needed.
+

We have not discussed benchmarking, which is another hard issue: who should you +compare with? where? how? For example, a global comparison between zlib and +decompress is not very relevant in many ways – especially because of the +garbage collector. This could be another article!

Finally, all of these optimizations should be done by flambda; the difference +between compiling decompress with or without flambda is not very big. We +optimized decompress by hand mostly to keep compatibility with OCaml (since +flambda needs another switch) and, in this way, to gain an understanding of +flambda optimizations so that we can use it effectively!

An introduction to fuzzing OCaml with AFL, Crowbar and Bun

tarides — 2019-09-04T00:00:00-00:00

American Fuzzy Lop or AFL is a fuzzer: a program that tries to find bugs in +other programs by sending them various auto-generated inputs. This article covers the +basics of AFL and shows an example of fuzzing a parser written in OCaml. It also introduces two +extensions: the Crowbar library which can be used to fuzz any kind of OCaml program or +function and the Bun tool for integrating fuzzing into your CI.

All of the examples given in this article are available on GitHub at +ocaml-afl-examples. The README contains all the information you need to understand, +build and fuzz them yourself.

What is AFL?

AFL actually isn't just a fuzzer but a set of tools. What makes it so good is that it doesn't just +blindly send random input to your program hoping for it to crash; it inspects the execution paths +of the program and uses that information to figure out which mutations to apply to the previous +inputs to trigger new execution paths. This approach allows for much more efficient and reliable +fuzzing (as it will try to maximize coverage) but requires the binaries to be instrumented so the +execution can be monitored.

AFL provides wrappers for the common C compilers that you can use to produce the instrumented +binaries along with the CLI fuzzing client: afl-fuzz.

afl-fuzz is straight-forward to use. It takes an input directory containing a few initial valid +inputs to your program, an output directory and the instrumented binary. It will then repeatedly +mutate the inputs and feed them to the program, registering the ones that lead to crashes or +hangs in the output directory.

Because it works in such a way, it makes it very easy to fuzz a parser.

To fuzz a parse.exe binary, that takes a file as its first command-line argument and parses it, +you can invoke afl-fuzz in the following way:

$ afl-fuzz -i inputs/ -o findings/ /path/to/parse.exe @@

The findings/ directory is where afl-fuzz will write the crashes it finds, it will create it +for you if it doesn't exist. +The inputs/ directory contains one or more valid input files for your +program. By valid we mean "that don't crash your program". +Finally the @@ part tells afl-fuzz where on the command line the input file should be passed to +your program, in our case, as the first argument.

Note that it is possible to supply afl-fuzz with more detail about how to invoke your program. If +you need to pass it command-line options for instance, you can run it as:

$ afl-fuzz -i inputs/ -o findings/ -- /path/to/parse.exe --option=value @@

If you wish to fuzz a program that takes its input from standard input, you can also do that by removing the +@@ from the afl-fuzz invocation.

Once afl-fuzz starts, it will draw a fancy looking table on the standard output to keep you +updated about its progress. From there, you'll mostly be interested in is the top right +corner which contains the number of crashes and hangs it has found so far:

+ + + + +

You might need to change some of your CPU settings to achieve better performance while fuzzing. +afl-fuzz's output will tell you if that's the case and guide you through the steps required to +make that happen.

Using AFL to fuzz an OCaml parser

First of all, if you want to fuzz an OCaml program with AFL you'll need to produce an instrumented +binary. afl-fuzz has an option to work with regular binaries but you'd lose a lot of what makes it +efficient. To instrument your binary you can simply install a +afl opam switch and build your +executable from there. AFL compiler variants are available from OCaml 4.05.0 onwards. To install such +a switch you can run:

$ opam switch create fuzzing-switch 4.07.1+afl

If your program already parses the standard input or a file given to it via the command line, you +can simply build the executable from your +afl switch and adapt the above examples. If it doesn't, +it's still easy to fuzz any parsing function.

Imagine we have a simple-parser library which exposes the following parse_int function:

val parse_int: string -> (int, [> `Msg of string]) result
+(** Parse the given string as an int or return [Error (`Msg _)].
+    Does not raise, usually... *)

We want to use AFL to make sure our function is robust and won't crash when receiving unexpected +inputs. As you can see the function returns a result and isn't supposed to raise exceptions. We want +to make sure that's true.

To find crashes, AFL traps the signals sent by your program. That means that it will consider +uncaught OCaml exceptions as crashes. That's good because it makes it really simple to write a +fuzz_me.ml executable that fits what afl-fuzz expects:

let () =
+  let file = Sys.argv.(1) in
+  let ic = open_in file in
+  let length = in_channel_length ic in
+  let content = really_input_string ic length in
+  close_in ic;
+  ignore (Simple_parser.parse_int content)

We have to provide example inputs to AFL so we can write a valid file to the inputs/ directory +containing 123 and an invalid file containing not an int. Both should parse without crashing +and make good starting point for AFL as they should trigger different execution paths.

Because we want to make sure AFL does find crashes we can try to hide a bug in our function:

let parse_int s =
+  match List.init (String.length s) (String.get s) with
+  | ['a'; 'b'; 'c'] -> failwith "secret crash"
+  | _ -> (
+      match int_of_string_opt s with
+      | None -> Error (`Msg (Printf.sprintf "Not an int: %S" s))
+      | Some i -> Ok i)

Now we just have to build our native binary from the right switch and let afl-fuzz do the rest:

$ afl-fuzz -i inputs/ -o findings/ ./fuzz_me.exe @@

It should find that the abc input leads to a crash rather quickly. Once it does, you'll see it in +the top right corner of its output as shown in the picture from the previous section.

At this point you can interrupt afl-fuzz and have a look at the content of the findings/crashes:

$ ls findings/crashes/
+id:000000,sig:06,src:000111,op:havoc,rep:16  README.txt

As you can see it contains a README.txt which will give you some details about the afl-fuzz +invocation used to find the crashes and how to reproduce them in the folder and a file of the form +id:...,sig:...,src:...,op:...,rep:... per crash it found. Here there's just one:

$ cat findings/crashes/id:000000,sig:06,src:000111,op:havoc,rep:16
+abc

As expected it contains our special input that triggers our secret crash. We can rerun the program +with that input ourselves to make sure it does trigger it:

$ ./fuzz_me.exe findings/crashes/id:000000,sig:06,src:000111,op:havoc,rep:16
+Fatal error: exception Failure("secret crash")

No surprise here, it does trigger our uncaught exception and crashes shamefully.

Using Crowbar and AFL for property-based testing

This works well but only being able to fuzz parsers is quite a limitation. That's where Crowbar +comes into play.

Crowbar is a property-based testing framework. It's much like Haskell's QuickCheck. +To test a given function, you define how its arguments are shaped, a set of properties the result +should satisfy and it will make sure they hold with any combinations of randomly generated +arguments. +Let's clarify that with an example.

I wrote a library called Awesome_list and I want to test its sort function:

val sort: int list -> int list
+(** Sorts the given list of integers. Result list is sorted in increasing
+    order, most of the time... *)

I want to make sure it really works so I'm going to use Crowbar to generate a whole lot of +lists of integers and verify that when I sort them with Awesome_list.sort the result is, well... +sorted.

We'll write our tests in a fuzz_me.ml file. +First we need to tell Crowbar how to generate arguments for our function. It exposes some +combinators to help you do that:

let int_list = Crowbar.(list (range 10))

Here we're telling Crowbar to generate lists of any size, containing integers ranging from 0 +to 10. Crowbar also exposes more complex and custom generator combinators so don't worry, +you can use it to build more complex arguments.

Now we need to define our property. Once again it's pretty simple, we just want the output to be +sorted:

let is_sorted l =
+  let rec is_sorted = function
+    | [] | [_] -> true
+    | hd::(hd'::_ as tl) -> hd <= hd' && is_sorted tl
+  in
+  Crowbar.check (is_sorted l)

All that's left to do now is to register our test:

let () =
+  Crowbar.add_test ~name:"Awesome_list.sort" [int_list]
+      (fun l -> is_sorted (Awesome_list.sort l))

and to compile that fuzz_me.ml file to a binary. Crowbar will take care of the magic.

We can run that binary in "Quickcheck" mode where it will either try a certain amount of random +inputs or keep trying until one of the properties breaks depending on the command-line options +we pass it. +What we're interested in here is its less common "AFL" mode. Crowbar made it so our executable +can be used with afl-fuzz just like that:

$ afl-fuzz -i inputs -o findings -- ./fuzz_me.exe @@

What will happen then is that our fuzz_me.exe binary will read the inputs provided by afl-fuzz +and use it to determine which test to run and how to generate the arguments to pass to our function. +If the properties are satisfied, the binary will exit normally; if they aren't, it will make sure +that afl-fuzz interprets that as a crash by raising an exception.

A nice side-effect of Crowbar's approach is that afl-fuzz will still be able to pick up +crashes. For instance, if we implement Awesome_list.sort as:

let sort = function
+  | [1; 2; 3] -> failwith "secret crash"
+  | [4; 5; 6] -> [6; 5; 4]
+  | l -> List.sort Pervasives.compare l

and use AFL and Crowbar to fuzz-test our function, it will find two crashes: one for the input +[1; 2; 3] which triggers a crash and one for [4; 5; 6] for which the is_sorted +property won't hold.

The content of the input files found by afl-fuzz itself won't be of much help as it needs to be +interpreted by Crowbar to build the arguments that were passed to the function to trigger the bug. +We can invoke the fuzz_me.exe binary ourselves on one of the files in findings/crashes +and the Crowbar binary will replay the test and give us some more helpful information about what +exactly is going on:

$ ./fuzz_me.exe findings/crashes/id\:000000\,sig\:06\,src\:000011\,op\:flip1\,pos\:5 
+Awesome_list.sort: ....
+Awesome_list.sort: FAIL
+
+When given the input:
+
+    [1; 2; 3]
+    
+the test threw an exception:
+
+    Failure("secret crash")
+    Raised at file "stdlib.ml", line 33, characters 17-33
+    Called from file "awesome-list/fuzz/fuzz_me.ml", line 11, characters 78-99
+    Called from file "src/crowbar.ml", line 264, characters 16-19
+    
+Fatal error: exception Crowbar.TestFailure
+$ ./fuzz_me.exe findings/crashes/id\:000001\,sig\:06\,src\:000027\,op\:arith16\,pos\:5\,val\:+7 
+Awesome_list.sort: ....
+Awesome_list.sort: FAIL
+
+When given the input:
+
+    [4; 5; 6]
+    
+the test failed:
+
+    check false
+    
+Fatal error: exception Crowbar.TestFailure

We can see the actual inputs as well as distinguish the one that broke the invariant from the one +that triggered a crash.

Using `bun` to run fuzz testing in CI

While AFL and Crowbar provide no guarantees they can give you confidence that your implementation +is not broken. Now that you know how to use them, a natural follow-up is to want to run fuzz tests +in your CI to enforce that level of confidence.

Problem is, AFL isn't very CI friendly. First it has this refreshing output that isn't going to look +great on your travis builds output and it doesn't tell you much besides that it could or couldn't find +crashes or invariant infrigements

Hopefully, like most problems, this one has a solution: +bun. +bun is a CLI wrapper around afl-fuzz, written in OCaml, that helps you get the best out of AFL +effortlessly. It mostly does two things:

The first is that it will run several afl-fuzz processes in parallel +(one per core by default). afl-fuzz starts with a bunch of deterministic steps. In my experience, +using parallel processes during this phase rarely proved very useful as they tend to find the same +bugs or slight variations of those bugs. It only achieves its full potential in the second phase of +fuzzing.

The second thing, which is the one we're the most interested in, is that bun provides a useful +and CI-friendly summary of what's going on with all the fuzzing processes so far. When one of them +finds a crash, it will stop all processes and pretty-print all of the bug-triggering inputs to help +you reproduce and debug them locally. See an example bun output after a crash was found:

Crashes found! Take a look; copy/paste to save for reproduction:
+1432	echo JXJpaWl0IA== | base64 -d > crash_0.$(date -u +%s)
+1433	echo NXJhkV8QAA== | base64 -d > crash_1.$(date -u +%s)
+1434	echo J3Jh//9qdGFiYmkg | base64 -d > crash_2.$(date -u +%s)
+1435	09:35.32:[ERROR]All fuzzers finished, but some crashes were found!

Using bun is very similar to using afl-fuzz. Going back to our first parser example, we can +fuzz it with bun like this:

$ bun --input inputs/ --output findings/ /path/to/parse.exe

You'll note that you don't need to provide the @@ anymore. bun assumes that it should pass the +input as the first argument of your to-be-fuzzed binary.

bun also comes with an alternative no-kill mode which lets all the fuzzers run indefinitely +instead of terminating them whenever a crash is discovered. It will regularly keep you updated on +the number of crashes discovered so far and when terminated will pretty-print each of them just like +it does in regular mode.

This mode can be convenient if you suspect your implementation may contain a lot of bugs and +you don't want to go through the whole process of fuzz testing it to only find a single bug.

You can use it in CI by running bun --no-kill via timeout. For instance:

timeout --preserve-status 60m bun --no-kill --input inputs --output findings ./fuzz_me.exe

will fuzz fuzz_me.exe for an hour no matter what happens. When timeout terminates bun, it will +provide you with a handful of bugs to fix!

Final words

I really want to encourage you to use those tools and fuzzing in general. +Crowbar and bun are fairly new so you will probably encounter bugs or find that it lacks a feature +you want but combined with AFL they make for very nice tools to effectively test +critical components of your OCaml code base or infrastructure and detect newly-introduced bugs. +They are already used accross the MirageOS ecosystem where it has been used to fuzz the TCP/IP stack +mirage-tcpip and the DHCP implementation charrua thanks to +somerandompacket. +You can consult Crowbar's hall of fame to find out about bugs uncovered by this +approach.

I also encourage anyone interested to join us in using this promising toolchain, report those bugs, +contribute those extra features and help the community build more robust software.

Finally if you wish to learn more about how to efficienly use fuzzing for testing I recommend the +excellent Write Fuzzable Code article by John Regehr.

Decompress: The New Decompress API

tarides — 2019-08-26T00:00:00-00:00

RFC 1951 is one of the most used standards. Indeed, +when you launch your Linux kernel, it inflates itself according zlib +standard, a superset of RFC 1951. Being a widely-used standard, we decided to +produce an OCaml implementation. In the process, we learned many lessons about +developing OCaml where we would normally use C. So, we are glad to present +decompress.

One of the many users of RFC 1951 is Git, which uses it to pack data +objects into a PACK file. At the request of @samoht, +decompress appeared some years ago as a Mirage-compatible replacement for zlib +to be used for compiling a MirageOS unikernel with +ocaml-git. Today, this little project passes a major release with +substantial improvements in several domains.

decompress provides an API for inflating and deflating flows[1]. The main +goal is to provide a platform-agnostic library: one which may be compiled on +any platform, including JavaScript. We surely cannot be faster than C +implementations like zstd or lz4, but we can play some +optimisation tricks to help bridge the gap. Additionally, OCaml can protect the +user against lot of bugs via the type-system and the runtime too (e.g. using +array bounds checking). ocaml-tls was implemented partly in +response to the famous failure of openssl; a vulnerability +which could not exist in OCaml.

[1]: A flow, in MirageOS land, is an abstraction which wants to receive +and/or transmit something under a standard. So it's usual to say a TLS-flow +for example.

API design

The API should be the most difficult part of designing a library - it reveals +what we can do and how we should do it. In this way, an API should:

constrain the user to avoid security issues; too much freedom can be a bad

thing. As an example, consider the Hashtbl.create function, which allows the +user to pass ~random:false to select a fixed hash function. The resulting +hashtable suffers deterministic key collisions, which can be exploited by an +attacker. +

+An example of good security-awareness in API design can be seen in +digestif, which provided an unsafe_compare instead of the common +compare function (before eqaf.0.5). In this way, it enforced the user to +create an alias of it if they want to use a hash in a Map – however, by this +action, they should know that they are not protected against a timing-attack.

allow some degrees of freedom to fit within many environments; a

constrained API cannot support a hostile context. For example, when compiling +to an ESP32 target, even small details such as the length of a stream +input buffer must be user-definable. When deploying to a server, memory +consumption should be deterministic. +

+Of course, this is made difficult when too much freedom will enable misuse of +the API – an example is dune which wants consciously to limit the user +about what they can do with it.

imply an optimal design of how to use it. Possibilities should serve the +user, but these can make the API harder to understand; this is why +documentation is important. Your API should tell your users how it wants to +be treated.

A dbuenzli API

From our experiences with protocol/format, one design stands out: the +dbuenzli API. If you look into some famous libraries in the OCaml +eco-system, you probably know uutf, jsonm or xmlm. All +of these libraries provide the same API for computing a Unicode/JSON/XML flow – +of course, the details are not the same.

From a MirageOS perspective, even if they use the in_channel/out_channel +abstraction rather than a Mirage flow, these libraries +are system-agnostic since they let the user to choose input and output buffers. +Most importantly, they don't use the standard OCaml Unix module, which cannot +be used in a unikernel.

The APIs are pretty consistent and try to do their best-effort[2] of +decoding. The design has a type state which represents the current system +status; the user passes this to decode/encode to carry out the processing. +Of course, these functions have a side-effect on the state internally, but +this is hidden from the user. One advantage of including states in a design is +that the underlying implementation is very amenable to compiler optimisations (e.g. +tail-call optimisation). Internally, of course, we have a porcelain[3] +implementation where any details can have an rational explanation.

In the beginning, decompress wanted to follow the same interface without the +mutability (a choice about performances) and it did. Then, the hard test was to +use it in a bigger project; in this case, ocaml-git. An iterative +process was used to determine what was really needed, what we should not provide +(like special cases) and what we should provide to reach an uniform API that is +not too difficult to understand.

From this experience, we finalised the initial decompress API and it did not +change significantly for 4 versions (2 years).

[2]: best-effort means an user control on the error branch where we don't +leak exception (or more generally, any interrupts)

[3]: porcelain means implicit invariants held in the mind of the programmer +(or the assertions/comments).

The new `decompress` API

The new decompress keeps the same inflation logic, but drastically changes the +deflator to make the flush operation clearer. For many purposes, people don't +want to hand-craft their compressed flows – they just want +of_string/to_string functions. However, in some contexts (like a PNG +encoder/decoder), the user should be able to play with decompress in detail +(OpenPGP needs this too in RFC 4880).

The Zlib format

Both decompress and zlib use Huffman coding, an algorithm +for building a dictionary of variable-length codewords for a given set of +symbols (in this case, bytes). The most common byte is assigned the shortest bit +sequence; less common bytes are assigned longer codewords. Using this +dictionary, we just translate each byte into its codeword and we should achieve +a good compression ratio. Of course, there are other details, such as the fact +that all Huffman codes are prefix-free. The compression can be +taken further with the LZ77 algorithm.

The zlib format, a superset of the RFC 1951 format, is easy +to understand. We will only consider the RFC 1951 format, since zlib adds only +minor details (such as checksums). It consists of several blocks: DEFLATE +blocks, each with a little header, and the contents. There are 3 kinds of +DEFLATE blocks:

a FLAT block; no compression, just a blit from inputs to the current block.
a FIXED block; compressed using a pre-computed Huffman code.
a DYNAMIC block; compressed using a user-specified Huffman code.

The FIXED block uses a Huffman dictionary that is computed when the OCaml runtime +is initialised. DYNAMIC blocks use dictionaries specified by the user, and so +these must be transmitted alongside the data (after being compressed with +another Huffman code!). The inflator decompresses this DYNAMIC dictionary and uses +it to do the reverse translation from bit sequences to bytes.

Inflator

The design of the inflator did not change a lot from the last version of +decompress. Indeed, it's about to take an input, compute it and return an +output like a flow. Of course, the error case can be reached.

So the API is pretty-easy:

val decode : decoder -> [ `Await | `Flush | `End | `Malformed of string ]

As you can see, we have 4 cases: one which expects more inputs (Await), the +second which asks to the user to flush internal buffer (Flush), the End case +when we reach the end of the flow and the Malformed case when we encounter an +error.

For each case, the user can do several operations. Of course, about the Await +case, they can refill the contents with an other inputs buffer with:

val src : decoder -> bigstring -> off:int -> len:int -> unit

This function provides the decoder a new input with len bytes to read +starting at off in the given bigstring.

In the Flush case, the user wants some information like how many bytes are +available in the current output buffer. Then, we should provide an action to +flush this output buffer. In the end, this output buffer should be given by +the user (how many bytes they want to allocate to store outputs flow).

type src = [ `Channel of in_channel | `Manual | `String of string ]
+
+val dst_rem : decoder -> int
+val flush : decoder -> unit
+val decoder : src -> o:bigstring -> w:bigstring -> decoder

The last function, decoder, is the most interesting. It lets the user, at the +beginning, choose the context in which they want to inflate inputs. So they +choose:

src, where come from inputs flow
o, output buffer
w, window buffer

o will be used to store inflated outputs, dst_rem will give to us how many +bytes inflator stored in o and flush will just set decoder to be able to +recompute the flow.

w is needed for lz77 compression. However, as we said, we let +the user give us this intermediate buffer. The idea behind that is to let the +user prepare an inflation. For example, in ocaml-git, instead of +allocating w systematically when we want to decompress a Git object, we +allocate w one time per threads and all are able to use it and re-use it. +In this way, we avoid systematic allocations (and allocate only once time) which +can have a serious impact about performances.

The design is pretty close to one idea, a description step by the decoder +function and a real computation loop with the decode function. The idea is to +prepare the inflation with some information (like w and o) before the main +(and the most expensive) computation. Internally we do that too (but it's mostly +about a macro-optimization).

It's the purpose of OCaml in general, be able to have a powerful way to describe +something (with constraints). In our case, we are very limited to what we need +to describe. But, in others libraries like angstrom, the description +step is huge (describe the parser according to the BNF) and then, we use it to +the main computation, in the case of angstrom, the parsing (another +example is [cmdliner][cmdliner]).

This is why decoder can be considered as the main function where decode can +be wrapped under a stream.

Deflator

The deflator is a new (complex) deal. Indeed, behind it we have two concepts:

the encoder (according to RFC 1951)
the compressor

For this new version of decompress, we decide to separate these concepts where +one question leads all: how to put my compression algorithm? (instead to use +LZ77).

In fact, if you are interested in compression, several algorithms exist and, in +some context, it's preferable to use lzwa for example or rabin's +fingerprint (with duff), etc.

Functor

The first idea was to provide a functor which expects an implementation of the +compression algorithm. However, the indirection of a functor comes with (big) +performance cost. Consider the following functor example:

module type S = sig
+  type t
+  val add : t -> t -> t
+  val one : t
+end
+
+module Make (S : S) = struct let succ x = S.add x S.one end
+
+include Make (struct
+  type t = int
+  let add a b = a + b
+  let one = 1
+end)
+
+let f x = succ x

Currently, with OCaml 4.07.1, the f function will be a caml_apply2. We might +wish for a simple inlining optimisation, allowing f to become an +addq instruction (indeed, flambda does this), but optimizing +functors is hard. As we learned from Pierre Chambart, it is possible +for the OCaml compiler to optimize functors directly, but this requires +respecting several constraints that are difficult to respect in practice.

Split encoder and compressor

So, the choice was done to made the encoder which respects RFC 1951 and the +compressor under some constraints. However, this is not what zlib did +and, by this way, we decided to provide a new design/API which did not follow, +in first instance, zlib (or some others implementations like +miniz).

To be fair, the choice from zlib and miniz comes from the first +point about API and the context where they are used. The main problem is the +shared queue between the encoder and the compressor. In C code, it can be hard +for the user to deal with it (where they are liable for buffer overflows).

In OCaml and for decompress, the shared queue can be well-abstracted and API +can ensure assumptions (like bounds checking).

Even if this design is much more complex than before, coverage tests are better +where we can separately test the encoder and the compressor. It breaks down the +initial black-box where compression was intrinsec with encoding – which was +error-prone. Indeed, decompress had a bug about generation of +Huffman codes but we never reached it because the (bad) +compressor was not able to produce something (a specific lengh with a specific +distance) to get it.

NOTE: you have just read the main reason for the new version of decompress!

The compressor

The compressor is the most easy part. The goal is to produce from an inputs +flow, an outputs flow which is an other (more compacted) representation. This +representation consists to:

A literal, the byte as is
A copy code with an offset and a length

The last one say to copy length byte(s) from offset. For example, aaaa can +be compressed as [ Literal 'a'; Copy (offset:1, len:3) ]. By this way, instead +to have 4 bytes, we have only 2 elements which will be compressed then by an +Huffman coding. This is the main idea of the lz77 +compression.

However, the compressor should need to deal with the encoder. An easy interface, +à la uutf should be:

val compress : state -> [ `Literal of char | `Copy of (int * int) | `End | `Await ]

But as I said, we need to feed a queue instead.

At this point, the purpose of the queue is not clear and not really explained. +The signature above still is a valid and understandable design. Then, we can +imagine passing Literal and Copy directly to the encoder. However, we should +(for performance purpose) use a delaying tactic between the compressor and the +deflator[^4].

Behind this idea, it's to be able to implement an hot-loop on the encoder +which will iter inside the shared queue and transmit/encode contents +directly to the outputs buffer.

So, when we make a new state, we let the user supply their queue:

val state : src -> w:bistring -> q:queue -> state
+val compress : state -> [ `Flush | `Await | `End ]

The Flush case appears when the queue is full. Then, we refind the w window +buffer which is needed to produce the Copy code. A copy code is limited +according RFC 1951 where offset can not be upper than the length of the window +(commonly 32ko). length is limited too to 258 (an arbitrary choice).

Of course, about the Await case, the compressor comes with a src function as +the inflator. Then, we added some accessors, literals and distances. The +compressor does not build the Huffman coding which needs +frequencies, so we need firstly to keep counters about that inside the state and +a way to get them (and pass them to the encoder).

[4]: About that, you should be interesting by the reason of why GNU yes is so +fast where the secret is just about buffering.

The encoder

Finally, we can talk about the encoder which will take the shared queue filled +by the compressor and provide an RFC 1951 compliant output flow.

However, we need to consider a special detail. When we want to make a +DYNAMIC block from frequencies and then encode the inputs flow, we can reach a +case where the shared queue contains an opcode (a literal or a copy) which +does not appear in our dictionary.

In fact, if we want to encode [ Literal 'a'; Literal 'b' ], we will not try to +make a dictionary which will contains the 256 possibilities of a byte but we +will only make a dictionary from frequencies which contains only 'a' and +'b'. By this way, we can reach a case where the queue contains an opcode +(like Literal 'c') which can not be encoded by the pre-determined +Huffman coding – remember, the DYNAMIC block starts with +the dictionary.

Another point is about inputs. The encoder expects, of course, contents from +the shared queue but it wants from the user the way to encode contents: which +block we want to emit. So it has two entries:

the shared queue
an user-entry

So for many real tests, we decided to provide this kind of API:

type dst = [ `Channel of out_channel | `Buffer of Buffer.t | `Manual ]
+
+val encoder : dst -> q:queue -> encoder
+val encode : encoder -> [ `Block of block | `Flush | `Await ] -> [ `Ok | `Partial | `Block ]
+val dst : encoder -> bigstring -> off:int -> len:int -> unit

As expected, we take the shared queue to make a new encoder. Then, we let the +user to specify which kind of block they want to encode by the Block +operation.

The Flush operation tries to encode all elements present inside the shared +queue according to the current block and feed the outputs buffer. From it, the +encoder can returns some values:

Ok and the encoder encoded all opcode from the shared queue
Partial, the outputs buffer is not enough to encode all opcode, the user +should flush it and give to us a new empty buffer with dst. Then, they must +continue with the Await operation.
Block, the encoder reachs an opcode which can not be encoded with the +current block (the current dictionary). Then, the user must continue with a new +Block operation.

The hard part is about the ping-pong game between the user and the encoder +where a Block expects a Block response from the user and a Partial expects +an Await response. But this design reveals something higher about zlib +this time: the flush mode.

The flush mode

Firstly, we talk about mode because zlib does not allow the user to +decide what they want to do when we reach a Block or a Ok case. So, it +defines some under-specified modes to apply a policy of what +to do in this case.

In decompress, we followed the same design and see that it may be not a good +idea where the logic is not very clear and the user wants may be an another +behavior. It was like a black-box with a black-magic.

Because we decided to split encoder and compressor, the idea of the flush mode +does not exists anymore where the user explicitly needs to give to the encoder +what they want (make a new block? which block? keep frequencies?). So we broke +the black-box. But, as we said, it was possible mostly because we can abstract +safely the shared queue between the compressor and the encoder.

OCaml is an expressive language and we can really talk about a queue where, in +C, it will be just an other array. As we said, the deal is about performance, +but now, we allow the user the possibility to write their code in this corner-case +which is when they reachs Block. Behaviors depends only on them.

APIs in general

The biggest challenge of building a library is defining the API - you must +strike a compromise between allowing the user the flexibility to express their +use-case and constraining the user to avoid API misuse. If at all possible, +provide an intuitive API: force the user not to need to think about security +issues, memory consumption or performance.

Avoid making your API so expressive that it becomes unusable, but beware that +this sets hard limits on your users: the current decompress API can be used to +build of_string / to_string functions, but the opposite is not true - you +definitely cannot build a stream API from of_string / to_string.

The best advice when designing a library is to keep in mind what you really +want and let the other details fall into place gradually. It is very important +to work in an iterative loop of repeatedly trying to use your library; only this +can highlight bad design, corner-cases and details.

Finally, use and re-use it on your tests (important!) and inside higher-level +projects to give you interesting questions about your design. The last version +of decompress was not used in ocaml-git mostly because the flush +mode was unclear.

i-Lab 2019

tarides — 2019-07-05T00:00:00-00:00

We are thrilled to announce that Tarides is laureate of the 21st +edition of the i-Lab innovation contest +for its innovative technological solution: OSMOSE.

Organized by the French Ministry of Higher Education, Research and +Innovation in partnership with Bpifrance, the objective of this +competition is to identify and support innovative technology-based +projects. This year, over 700 applications have been registered and 75 +projects rewarded. The jury was composed of fifty experts (business +leaders, researchers, former laureate, start-uppers, engineers, +consultants, investors) and was chaired by Ludovic Le Moan, CEO of +Sigfox.

The OSMOSE solution is a software infrastructure platform to deploy +secure and distributed IoT applications, using low-resource +constraints and providing low-latency performance. This platform is +built upon innovative and open-source projects (in particular MirageOS +and Irmin) which were started at the University of Cambridge, over 10 +years ago, where the founders of Tarides met. Tarides uses unikernel +technologies and applies the research done in programming languages to +real-world systems to build safe and performant applications +specialized to their runtime environment.

If you are interested by the project, contact us: +contact@tarides.com

Or check our position paper +to learn more about it.

Release of OCamlFormat 0.10

tarides — 2019-06-27T00:00:00-00:00

We are pleased to announce the release of OCamlFormat 0.10 (available on opam).

There have been numerous changes since the last release, so here is a comprehensive list of the new features and breaking changes to help the transition from OCamlFormat 0.9.

ocamlformat-0.10 now works on the 4.08 AST, although the formatting should not differ greatly from the one of ocamlformat-0.9 in this regard. +Please note that it is necessary to build ocamlformat with 4.08 to be able to parse new features like let*.

Upgrading from ocamlformat-0.9 requires to install the following dependencies:

ocaml-migrate-parsetree >= 1.3.1 (upgrade)
uuseg >= 10.0.0 (new)
uutf >= 1.0.1 (upgrade)

This release focuses on preserving the style of the original source and on handling more ocp-indent options.

Style preservation

Expression grouping

The new option exp-grouping has been added to preserve the keywords begin/end that are used to delimit expressions instead of parentheses. exp-grouping=parens always uses parentheses to delimit expressions. exp-grouping=preserve preserves the original grouping syntax (parentheses or begin/end).

Horizontal alignment

Horizontal alignment is something that users often use to make pattern-matching or type declarations easier to read, and it is a feature that has been requested many times. Three new options have been added to horizontally align the lines.

align-cases horizontally aligns the match/try cases:

let fooooooooooo =
+  match foooooooooooooooooooooooo with
+  | Bfooooooooooooooooo -> foooooooooooo
+  | C (a, b, c, d)      -> fooooooooooooooooooo
+  | _                   -> fooooooooooooooooooo

align-constructors-decl horizontally aligns type declarations:

type t =
+  | ( :: ) of a * b
+  | []     of looooooooooooooooooooooooooooooooooooooong_break

align-variants-decl horizontally aligns variants type declarations:

type x =
+  [ `Foooooooo      of int
+  | `Fooooooooooooo of int ]

Preserve blank lines in sequences

The new option sequence-blank-line decides whether a blank line is preserved between expressions of a sequence. sequence-blank-line=compact will not keep any blank line between expressions of a sequence, this is still the default behavior. sequence-blank-line=preserve will keep a blank line between two expressions of a sequence if the input contains at least one.

This option can help preserving the readability of the code in this situation:

let foo x y =
+  do_some_setup y ;
+
+  important_function x

Supporting more `ocp-indent` options

The long term goal of ocamlformat is to handle every ocp-indent option, this release got closer to this goal as the following ocp-indent options are now supported by ocamlformat:

max_indent
with
strict_with
ppx_stritem_ext
base
in
type

Offset added to a new line

The new option max-indent sets the maximum offset (number of columns) added to a new line in addition to the offset of the previous line. If this offset is set to 2 columns, then each new line can only be indented by 2 columns more in addition to the previous line, for example:

let () =
+  fooooo
+  |> List.iter (fun x ->
+    let x = x $ y in
+    fooooooooooo x)

This option is equivalent to the max_indent option of ocp-indent, and it will be set if max_indent is set in an .ocp-indent configuration file.

Indentation of pattern matching cases

The new options funtion-indent and match-indent respectively decide the indentation of function cases and the indentation of match/try cases. +These options are equivalent to the with option of ocp-indent, and they will be set if with is set in an ocp-indent configuration file. +If the indentation is set to 4 columns, cases are formatted like this:

let foooooooo = function
+    | fooooooooooooooooooooooo -> foooooooooooooooooooooooooo
+
+let foooooooo =
+  match fooooooooooooooooooooooo with
+      | fooooooooooooooooooooooo -> foooooooooooooooooooooooooo

The new options function-indent-nested and match-indent-nested respectively decide whether the function-indent and the match-indent parameters should be applied even when in a sub-block. If these options are set to never, it only applies function-indent or match-indent if the function or match block starts a line. If these options are set to always, then the indent parameters are always applied. The auto value applies the indentation parameter when seen fit.

These options are equivalent to the strict_with option of ocp-indent, and they will be set if strict_with is set in an ocp-indent configuration file.

Indentation inside extension nodes

The new option extension-indent sets the indentation of items (that are not at structure level) inside extension nodes. +The new option stritem-extension-indent sets the indentation of structure items inside extension nodes. This option is equivalent to the ppx_stritem_ext option of ocp-indent, and it will be set if ppx_stritem_ext is set in an .ocp-indent configuration file.

For example if extension-indent is set to 5 and stritem-extension-indent is set to 3:

let foo =
+  [%foooooooooo
+       fooooooooooooooooooooooooooo foooooooooooooooooooooooooooooooooo
+         foooooooooooooooooooooooooooo]
+  [@@foooooooooo
+       fooooooooooooooooooooooooooo foooooooooooooooooooooooooooooooooo
+         foooooooooooooooooooooooooooo]
+
+[@@@foooooooooo
+   fooooooooooooooooooooooooooo foooooooooooooooooooooooooooooooooo
+     foooooooooooooooooooooooooooo]

Let-binding indentation

The new option let-binding-indent sets the indentation of let binding expressions if they do not fit on a single line. This option is equivalent to the base option of ocp-indent. +The new option indent-after-in sets the indentation after let ... in, unless followed by another let. This option is equivalent to the in option of ocp-indent. +The new option type-decl-indent sets the indentation of type declarations if they do not fit on a single line. This option is equivalent to the type option of ocp-indent.

These options will be set if their ocp-indent counterparts are set in an .ocp-indent configuration file.

Miscellaneous features

This release also brings some new options, new values for existing features, or corrects erroneous behaviours.

Indicate multiline delimiters

The former indicate-multiline-delimiters boolean option is now a 3-valued option:

indicate-multiline-delimiters=space (was equivalent to true) prints a space inside the delimiter to indicate the matching one is on a different line.
indicate-multiline-delimiters=no (was equivalent to false) doesn't do anything special to indicate the closing delimiter.
indicate-multiline-delimiters=closing-on-separate-line is the new feature of this option, it makes sure that the closing delimiter is on its own line.

On this example we can see the closing parenthesis delimiting the nested pattern-matchings are on their own line and are aligned with the matching opening parenthesis:

let () =
+   match v with
+   | None -> None
+   | Some x ->
+       ( match x with
+       | None -> None
+       | Some x ->
+           ( match x with
+           | None -> None
+           | Some x -> x
+           )
+       )

Formatting of literal strings

break-string-literals=newlines now takes into account pretty-printing commands like @,, @; and @\n to produce more readable strings. A new value for this option has been added, break-string-literals=newlines-and-wrap, to break lines at newlines delimiters (including pretty-printing commands) and also wrap the string literals at the margin.

Here is how break-string-literals=newlines-and-wrap formats a string:

let fooooooooooo =
+  "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod \
+   tempor incididunt ut labore et dolore magna aliqua.@;\
+   Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi \
+   ut aliquip ex ea commodo consequat.@;\
+   Duis aute irure dolor in reprehenderit in voluptate velit esse cillum \
+   dolore eu fugiat nulla pariatur.@;\
+   Excepteur sint occaecat cupidatat non proident, sunt in culpa qui \
+   officia deserunt mollit anim id est laborum."

Warning: the break-string-literals will likely be removed in the next release and the default behavior would be newlines-and-wrap.

Break before the `in` keyword

The new option break-before-in has been added to decide whether the line should break before the in keyword of a let binding. break-before-in=fit-or-vertical will always break the line before the in keyword if the whole let binding does not fit on a single line, it is still the default behavior. break-before-in=auto will only break the line if the in keyword does not fit on the previous line.

For example:

let _ =
+  let short = this is short in
+  let fooo =
+    (this is very long) but (the in keyword can fit) on the same line in
+  foooooo

Indentation of nested pattern-matching

The new option nested-match defines the style of pattern-matchings nested in the last case of another pattern-matching. nested-match=wrap wraps the nested pattern-matching with parentheses and adds indentation, this is still the default behavior. nested-match=align vertically aligns the nested pattern-matching under the encompassing pattern-matching, for example:

let () =
+  match v with
+  | None -> None
+  | Some x ->
+  match x with
+  | None -> None
+  | Some x -> x

The new option cases-matching-exp-indent decides the indentation of cases right-hand sides which are match or try expressions. cases-matching-exp-indent=compact forces an indentation of 2, unless nested-match is set to align and this is the last case of the pattern matching. compact is the default behavior. cases-matching-exp-indent=normal indents as it would any other expression.

Whitelist of files to format

A new kind of configuration files is now handled by ocamlformat: .ocamlformat-enable files. +If the disable option is set, an .ocamlformat-enable file can list the files that ocamlformat should format even when the disable option is set. Each line in an .ocamlformat-enable file specifies a filename relative to the directory containing the .ocamlformat-enable file.

The .ocamlformat-enable files are using the same syntax as the .ocamlformat-ignore files: lines starting with # are ignored and can be used as comments.

These new configuration files do not contradict the existing .ocamlformat-ignore files, as .ocamlformat-enable are only considered when disable is set, and .ocamlformat-ignore are only considered when disable is not set.

Disable outside detected project

The disable-outside-detected-project option is now set by default.

When the option --enable-outside-detected-project is not set, .ocamlformat files outside of the project (including the one in XDG_CONFIG_HOME) are not read. The project root of an input file is taken to be the nearest ancestor directory that contains a .git or .hg or dune-project file. If no config file is found, formatting is disabled.

Space around collection-expressions

The former option space-around-collection-expressions that was deciding whether a space should be added inside the delimiters of collection expressions (lists, arrays, records, variants) has been replaced by 4 new options: space-around-arrays, space-around-lists, space-around-records and space-around-variants, to allow a finer grain customization.

Fit-or-vertical mode for pattern matching

The break-cases option that decides the shape of pattern matching has a new value fit-or-vertical. break-cases=fit-or-vertical tries to fit all or-patterns on the same line, otherwise breaks each or-pattern (they are wrapped in other modes). +For example if this set of or-patterns does not fit on a single line, we get the following output:

let ffffff =
+  match foooooooooooo with
+  | Aaaaaaaaaaaaaaaaa
+  | Bbbbbbbbbbbbbbbbb
+  | Ccccccccccccccccc
+  | Ddddddddddddddddd
+  | Eeeeeeeeeeeeeeeee -> foooooooooooooooooooo
+  | Fffffffffffffffff -> fooooooooooooooooo

K&R style for if-then-else

The if-then-else option now has a new value k-r that uses parentheses (when necessary) to reproduce a formatting close to the K&R style. For example:

let _ =
+  if b then (
+    something loooooooooooooooooooooooooooooooong enough to_trigger a break ;
+    this is more
+  ) else if b1 then (
+    something loooooooooooooooooooooooooooooooong enough to_trigger a break ;
+    this is more
+  ) else
+    e

Breaking changes

the indicate-multiline-delimiters option is no longer a boolean option but now has 3 values: space, no and closing-on-separate-line that are detailed in this patch note.
the disable-outside-detected-project option is now set by default.
the default preset profile has been removed (it was equivalent to the ocamlformat profile with break-cases=fit).
the space-around-collection-expressions option has been replaced by 4 new options: space-around-arrays, space-around-lists, space-around-records and space-around-variants.

What's next?

We strongly encourage our users to try out the conventional preset profile, as we plan to make it the default profile in a future release. This profile's purpose is to reproduce the most commonly encountered styles, and it may be more pleasing to the eye than the current default options.

As stated previously, the break-string-literals will likely be removed in the next release and the default behavior would be newlines-and-wrap.

Credits

This release also contains many other changes and bug fixes that we cannot detail here.

We would like to thank our maintainers and contributors for this release: Jules Aguillon, Josh Berdine, Hugo Heuzard, Guillaume Petiot and Thomas Refis, and especially our industrial users Jane Street, Ahrefs and Nomadic Labs that made this work possible by funding this project and providing helpful contributions and feedback.

We would be happy to provide support for more customers, please contact us at contact@tarides.com

If you wish to get involved with OCamlFormat development or file an issue, please read the contributing guide, any contribution is welcomed.

On the road to Irmin v2

tarides — 2019-05-13T00:00:00-00:00

Over the past few months, we have been heavily engaged in release +engineering the Irmin 2.0 release, +which covers multiple years of work on all of its constituent +elements. We first began Irmin in late 2013 to act as a +Git-like distributed and branchable storage substrate +that would let us escape the perils of POSIX filesystems.

The Irmin libraries provide snapshotting, branching and merging +operations over storage and can communicate via Git both on-disk and +remotely. Irmin today therefore consists of many discrete OCaml +libraries that compose together to form a set of mergeable data structures +that can be used in MirageOS unikernels and normal OCaml daemons such +as Tezos.

In this blog post, we wanted to explain some of the release +engineering ongoing, and to highlight some areas where we could use +help from the community to test out pieces (and hopefully find your +own uses in your own infrastructure for it). The overall effort is +tracked in mirage/irmin#658, so +feel free to comment on there as well.

ocaml-git

Irmin is parameterised over the exact communication mechanisms it uses +between nodes, both as an on-disk format and also the remoting +protocol. The most important concrete implementation is Git, which +has turned into the world’s most popular version control system. In +order to seamlessly integrate with Irmin, we embarked on an effort to +build a complete re-implementation of +Git from scratch in pure OCaml.

You can read details of the git 2.0 release +on this blog, but from a release engineering perspective we have steadily +been fixing corner cases in this implementation. The development +ocaml-git trees feature fixes to https+git, +for listing remotes, supporting +authenticated URIs and +more.

These fixes are possible because users tried end-to-end usecases that +found these corner cases, so we’d really like to see more. For +example, our friends at Robur have submitted fixes +from their integration of it into their upcoming CalDAV engine. +The Mirage canopy blog engine can now also +push/pull reliably from pure MirageOS unikernels between nodes, which +is a huge step.

If you get a chance to try ocaml-git in your infrastructure, please +let us know how you get along as we prepare a release of the git +libraries with all these fixes (which will be used in Irmin 2.0).

Wodan

Tezos and irmin-pack

The irmin-pack backend is +currently being reviewed and integrated ahead of Irmin 2.0 to provide +a significant improvement in disk usage -- more information to come soon. +There is a corresponding Tezos branch +using the Irmin 2.0 code that will be integrated downstream in Tezos +once we complete the Irmin 2.0 tests.

Irmin-GraphQL and “browser Irmin”

Another new area of huge interest to us is +GraphQL in order to provide frontends a rich +query language for Irmin hosted applications. Irmin 2.0 includes a +builtin GraphQL server so you can manipulate your Git repo via +GraphQL.

This post is just the precursor to the Irmin 2.0 release, so expect to +hear more about it in the coming weeks and months. This is primarily +a call for help from early adopters interested in helping the project +out. All of our code is liberally licensed open source, and so this +is a good time to tie together end-to-end usecases and help ensure we +don’t make any decisions in Irmin 2.0 that go counter to some product +you’d like to build. That’s only possible with your feedback, so +either get in touch via the issue tracker, on +discuss.ocaml.org via the mirageos tag, +or just email us.

A huge thank you to all our commercial customers, end users and open +source developers who have contributed their time, expertise and +financial support to help us achieve our goal of delivering a modern +storage stack in the spirit of Git. We look forward to getting Irmin +2.0 into your hands very soon!

An introduction to OCaml PPX ecosystem

tarides — 2019-05-09T00:00:00-00:00

These last few months, I spent some time writing new OCaml PPX rewriters or contributing to existing +ones. It's a really fun experience. Toying around with the AST taught me a lot about a language I +thought I knew really well. Turns out I actually had no idea what I was doing all these years.

All jokes aside, I was surprised that the most helpful tricks I learned while writing PPX rewriters +weren't properly documented. There already exist a few very good introduction articles on the +subject, like that +2014's article from Whitequark, +this more recent one from Rudi Grinberg +or even this last one from Victor Darvariu +I only discovered after I actually started writing my own. I still felt like they were slightly +outdated or weren't answering all the questions I had when I started playing with PPX and writing my +first rewriters.

I decided to share my PPX adventures in the hope that it can help others familiarize with this bit +of the OCaml ecosystem and eventually write their first rewriters. The scope of this article is not to +cover every single detail about the PPX internals but just to give a gentle introduction to +beginners to help them get settled. That also means I might omit things that I don't think are worth +mentioning or that might confuse the targetted audience but feel free to comment if you believe +this article missed an important point.

It's worth mentioning that a lot of the nice tricks mentioned in these lines were given to me by a +wonderful human being called Étienne Millon, thanks Étienne!

What is a PPX?

PPX rewriters or PPX-es are preprocessors that are applied to your code before passing it on to the +compiler. They don't operate on your code directly but on the Abstract Syntax Tree or AST resulting +from its parsing. That means that they can only be applied to syntactically correct OCaml code. You +can think of them as functions that take an AST and return a new AST.

That means that in theory you can do a lot of things with a PPX, including pretty bad and cryptic +things. You could for example replace every instance of true by false, swap the branches of any +if-then-else or randomize the order of every pattern-matching case. +Obviously that's not the kind of behaviour that we want as it would make it impossible to +understand the code since it would be so far from the actual AST the compiler would get. +In practice PPX-es have a well defined scope and only transform parts you explicitly annotated.

Understanding the OCaml AST

First things first, what is an AST. An AST is an abstract representation of your code. As the name +suggests it has a tree-like structure where the root describes your entire file. It has children for +each bits such as a function declaration or a type definition, each of them having their own +children, for example for the function name, its argument and its body and that goes on until you +reach a leaf such as a literal 1, "abc" or a variable for instance. +In the case of OCaml it's a set of recursive types allowing us to represent OCaml code as an OCaml +value. This value is what the parser passes to the compiler so it can type check and compile it to +native or byte code. +Those types are defined in OCaml's Parsetree module. The entry points there are the structure +type which describes the content of an .ml file and the signature type which describes the +content of an .mli file.

As mentionned above, a PPX can be seen as a function that transforms an AST. Writing a PPX thus +requires you to understand the AST, both to interpret the one you'll get as input and +to produce the right one as output. This is probably the trickiest part as unless you've already +worked on the OCaml compiler or written a PPX rewriter, that will probably be the first time you two +meet. Chances are also high that'll be a pretty bad first date and you will need some to time +to get to know each other.

The Parsetree module documentation, +is a good place to start. The above mentioned structure and signature types are at the root of +the AST but some other useful types to look at at first are:

expression which describes anything in OCaml that evaluates to a value, the right hand side of a +let binding for instance.
pattern which is what you use to deconstruct an OCaml value, the left hand side of a let +binding or a pattern-matching case for example.
core_type which describes type expressions ie what you would find on the right hand side of a +value description in a .mli, ie val f : <what_goes_there>.
structure_item and signature_item which describe the top level AST nodes you can find in a +structure or signature such as type definitions, value or module declarations.

Thing is, it's a bit a rough and there's no detailed explanation about how a specific bit of code is +represented, just type definitions. Most of the time, the type, field, and variant names are +self-explanatory but it can get harder with some of the more advanced language features. +It turns out there are plenty of comments that are really helpful in the actual parsetree.mli file +and that aren't part of the generated documentation. You can find them on +github but I personally prefer to +have it opened in a VIM tab when I work on a PPX so I usually open +~/.opam/<current_working_switch>/lib/ocaml/compiler-libs/parsetree.mli.

This works well while exploring but you might also want a more straightforward approach to +discovering what the AST representation is for some specific OCaml code. The +ppx_tools opam package comes with a dumpast binary +that pretty prints the AST for any given piece of valid OCaml code. You can install it using opam:

$opam install ppx_tools

and then run it using ocamlfind:

$ocamlfind ppx_tools/dumpast some_file.ml

You can use it on .ml and .mli files or to quickly get the AST for an expression with the -e +option:

$ocamlfind ppx_tools/dumpast -e "1 + 1"

Similarly, you can use the -t or -p options to respectively pretty print ASTs from type +expressions or patterns.

Using dumpast to get both the ASTs of a piece of code using your future PPX and the resulting +preprocessed code is a good way to start and will help you figure out what are the steps required to +get there.

Note that you can use the compiler or utop have a similar feature with the -dparsetree flag. +Running ocamlc/ocamlopt -dparsetree file.ml will pretty print the AST of the given file while +running utop -dparsetree will pretty print the AST of the evaluated code alongside it's +evaluation. +I tend to prefer the pretty printed AST from dumpast but any of these tools will prove helpful +in understanding the AST representation of a given piece of OCaml code.

Language extensions interpreted by PPX-es

OCaml 4.02 introduced syntax extensions meant to be used by external tools such as PPX-es. Knowing +their syntax and meaning is important to understand how most of the existing rewriters +work because they usually look for those language extensions in the AST to know which part of it +they need to modify.

The two language extensions we're interested in here are extension nodes and attributes. They are +defined in detail in the OCaml manual (see the +attributes and +extension nodes sections) but I'll +try to give a good summary here.

Extension nodes are used in place of expressions, module expressions, patterns, type expressions or +module type expressions. Their syntax is [%extension_name payload]. We'll come back to the payload +part a little later. +You can also find extension nodes at the top level of modules or module signatures with the syntax +[%%extension_name payload]. +Hopefully the following cheatsheet can help you remember the basics of how and where you can use +them:

type t =
+  { a : int
+  ; b : [%ext pl]
+  }
+
+let x =
+  match 1 with
+  | 0 -> [%ext pl]
+  | [%ext pl] -> true
+
+[%%ext pl]

Because extension nodes stand where regular AST nodes should, the compiler won't accept them and +will give you an Uninterpreted extension error. Extension nodes have to be expanded by a PPX for +your code to compile.

Attributes are slightly different although their syntax is very close to extensions. Attributes +are attached to existing AST nodes instead of replacing them. That means that they don't necessarily +need to be transformed and the compiler will ignore unknown attributes by default. +They can come with a payload just like extensions and use @ instead of %. The number of @ +preceding the attribute name specifies which kind of node they are attached to:

let a = 12 [@attr pl]
+
+let b = "some string" [@@attr pl]
+
+[@@@attr pl]

In the first example, the attribute is attached to the expression 12 while in the second example +it is attached to the whole let b = "some string" value binding. The third one is of a slightly +different nature as it is a floating attribute. It's not attached to anything per-se and just ends +up in the AST as a structure item. +Because there is a wide variety of nodes to which you can attach attributes, I won't go too far into +details here but a good rule of thumb is that you use @@ attributes when you want them attached to +structure or signature items, for anything deeper within the AST structure such as patterns, +expressions or core types, use the single @ syntax. Looking at the Parsetree documentation can +help you figure out where you can find attributes.

Now let's talk about those payloads I mentioned earlier. You can think of them as "arguments" to +the extension points and attributes. You can pass different kinds of arguments and the syntax varies +for each of them:

let a = [%ext expr_or_str_item] 
+let b = [%ext: type_expr_or_sig_item]
+let c = [%ext? pattern]

As suggested in the examples, you can pass expressions or structure items using a space character, +type expressions or signature items (anything you'd find at the top level of a module signature) +using a : or a pattern using a ?.

Attributes' payload use the same syntax:

let a = 'a' [@attr expr_or_str_item]
+let b = 'b' [@attr: type_expr_or_sig_item]
+let a = 'a' [@attr? pattern]

Some PPX-es rely on other language extensions such as the suffix character you can attach to int +and float literals (10z could be used by a PPX to turn it into Z.of_string "10" for instance) +or quoted strings with a specific identifier ({ppx_name|some quoted string|ppx_name} can be used +if you want your PPX to operate on arbitrary strings and not only syntactically correct OCaml) but +attributes and extensions are the most commonly used ones.

Attributes and extension points can be expressed using an infix syntax. The attribute version is +barely used but some forms of the infix syntax for extension points are used by popular PPX-es and +it is likely you will encounter some of the following:

let infix_let_extension =
+  let%ext x = 2 in
+  ...
+
+let infix_match_extension =
+  match%ext y with ...
+
+let infix_try_extension =
+  try%ext f z with _ -> ...

which are syntactic sugar for:

let infix_let_extension =
+  [%ext let x = 2 in ...]
+
+let infix_match_extension =
+  [%ext match y with ...]
+
+let infix_try_extension =
+  [%ext try f z with _ -> ...]

A good example of a PPX making heavy use of these if +lwt_ppx. The OCaml manual also contains more examples +of the infix syntax in the Attributes and Extension points sections mentioned above.

The two main kind of PPX-es

There is a wide variety of PPX rewriters but the ones you'll probably see the most are Extensions and +Derivers.

Extensions

Extensions will rewrite tagged parts of the AST, usually extension nodes of the form +[%<extension_name> payload]. They will replace them with a different AST node of the same nature ie +if the extension point was located where an expression should be, the rewriter will produce an +expression. Good examples of extensions are:

ppx_getenv2 which replaces [%getenv SOME_VAR] with +the value of the environment variable SOME_VAR at compile time.
ppx_yojson which allows you to write Yojson values +using OCaml syntax to mimic actual json. For instance you'd use [%yojson {a = None; b = 1}] to +represent {"a": null, "b": 1} instead of the Yojson's notation: +Assoc [("a", Null); ("b", Int 1)].

Derivers

Derivers or deriving plugins will "insert" new nodes derived from type definitions annotated with a +[@@deriving <deriver_name>] attribute. They have various applications but are particularly useful +to derive functions that are tedious and error prone to write by hand such as comparison functions, +pretty printers or serializers. It's really convenient as you don't have to update those functions +every time you update your type definitions. They were inspired by Haskell Type classes. Good +examples of derivers are:

ppx_deriving itself comes with a bunch of deriving +plugins such as eq, ord or show which respectively derives, as you might have guessed, +equality, comparison and pretty-printing functions.
ppx_deriving_yojson which derives JSON +serializers and deserializers.
ppx_sexp_conv which derives s-expressions +converters.

Derivers often let you attach attributes to specify how some parts of the AST should be handled. For +example when using ppx_deriving_yojson you can use [@default some_val] to make a field of an +object optional:

type t =
+  { a: int
+  ; b: string [@default ""]
+  }
+[@@deriving of_yojson]

will derive a deserializer that will convert the JSON value {"a": 1} to the OCaml +{a = 1; b = ""}

How to write a PPX using `ppxlib`

Historically there was a few libraries used by PPX rewriter authors to write their PPX-es, including +ppx_tools and ppx_deriving but as the eco-system evolved, ppxlib emerged and is now the most +up-to-date and maintained library to write and handle PPX-es. It wraps the features of those +libraries in a single one. +I encourage you to use ppxlib to write new PPX-es as it is also easier to make various rewriters +work together if they are all registered through ppxlib and the PPX ecosystem would gain from +being unified around a single PPX library and driver.

It is also a great library and has some really powerful features to help you write your extensions +and derivers.

Writing an extension

The entry point of ppxlib for extensions is Ppxlib.Extension.declare. You have to use that +function to build an Extension.t, from which you can then build a Context_free.Rule.t before +registering your transformation so it's actually applied.

The typical my_ppx_extension.ml will look like:

open Ppxlib
+
+let extension =
+  Extension.declare
+    "my_extension"
+    some_context
+    some_pattern
+    expand_function
+
+let rule = Context_free.Rule.extension extension
+
+let () =
+  Driver.register_transformation ~rules:[rule] "my_transformation"

To compile it as PPX rewriter you'll need to put the following in your dune file:

(library
+ (public_name my_ppx)
+ (kind ppx_rewriter)
+ (libraries ppxlib))

Now let's go back a little and look at the important part:

let extension =
+  Extension.declare
+    "my_extension"
+    some_context
+    some_pattern
+    expand_function

Here "my_extension" is the name of your extension and that define how you're going to invoke it +in your extension point. In other words, to use this extension in our code we'll use a +[%my_extension ...] extension point.

some_context is a Ppxlib.Extension.Context.t and describes where this extension can be found in +the AST, ie can you use [%my_extension ...] as an expression, a pattern, a core type. The +Ppxlib.Extension.Context module defines a constant for each possible extension context which you +can pass as some_context. +This obviously means that it also describes the type of AST node to which it must be converted and +this property is actually enforced by the some_pattern argument. But we'll come back to that +later.

Finally expand_function is our actual extension implementation, which basically takes the payload, +a loc argument which contains the location of the expanded extension point, a path argument +which is the fully qualified path to the expanded node (eg. "file.ml.A.B") and returns the +generated code to replace the extension with.

Ast_pattern

Now let's get back to that some_pattern argument.

This is one of the trickiest parts of ppxlib to understand but it's also one its most +powerful features. The type for Ast_pattern is defined as ('a, 'b, 'c) t where 'a is +the type of AST nodes that are matched, 'b is the type of the values you're extracting from the +node as a function type and 'c is the return type of that last function. This sounded really +confusing to me at first and I'm guessing it might do to some of you too so let's give it a bit of +context.

Let's look at the type of Extension.declare:

val declare :
+  string ->
+  'context Context.t ->
+  (payload, 'a, 'context) Ast_pattern.t ->
+  (loc:Location.t -> path:string -> 'a) ->
+  t

Here, the expected pattern first type parameter is payload which means we want a pattern that +matches payload AST nodes. That makes perfect sense since it is used to describe what your +extension's payload should look like and what to do with it. +The last type parameter is 'context which again seems logical. As I mentioned earlier our +expand_function should return the same kind of node as the one where the extension was found. +Now what about 'a. As you can see, it describes what comes after the base loc and path +parameters of our expand_function. From the pattern point of view, 'a describes the parts of the +matched AST node we wish to extract for later consumption, here by our expander.

Ast_pattern contains a whole bunch of combinators to let you describe what your pattern should match +and a specific __ pattern that you must use to capture the various parts of the matched nodes. +__ has type ('a, 'a -> 'b, 'b) Ast_pattern.t which means that whenever it's used it changes the +type of consumer function in the returned pattern.

Let's consider a few examples to try wrapping our heads around this. Say I want to write an +extension that takes an expression as a payload and I want to pass this expression to my expander so +I can generate code based on its value. I can declare the extension like this:

let extension =
+  Extension.declare
+    "my_extension"
+    Extension.Context.expression
+    Ast_pattern.(single_expr_payload __)
+    expand_function

In this example, Extension.Context.expression has type expression Extension.Context.t, the +pattern has type (payload, expression -> expression, expression) Ast_pattern.t. The pattern says we +want to allow a single expression in the payload and capture it. If we decompose it a bit, we can +see that single_expr_payload has type +(expression, 'a, 'b) Ast_pattern.t -> (payload, 'a, 'b) Ast_pattern.t and is passed __ which +makes it a (expression, expression -> 'b, 'b) Ast_pattern.t and that's exactly what we want here +as our expander will have type loc: Location.t -> path: string -> expression -> expression!

It works similarly to Scanf.scanf when you think about it. Changing the pattern changes the type of the +consumer function the same way changing the format string does for Scanf functions.

This was a bit easy since we had a custom combinator just for that purpose so let's take a few more +complex examples. Now say we want to only allow pairs of integer and string constants expressions in +our payload. Instead of just capturing any expression and dealing with the error cases in the +expand_function we can let Ast_pattern deal with that and pass an int and string along to +our expander:

Ast_pattern.(single_expr_payload (pexp_tuple ((eint __)^::(estring __)^::nil)))

This one's a bit more elaborate but the idea is the same, we use __ to capture the int and string +from the expression and use combinators to specify that the payload should be made of a pair and +that gives us a: (payload, int -> string -> 'a, 'a) Ast_pattern.t which should be used with a +loc: Location.t -> path: string -> int -> string -> expression expander.

We can also specify that our extension should take something else than an expression as a payload, +say a pattern with no when clause so that it's applied as [%my_ext? some_pattern_payload]:

Ast_pattern.(ppat __ none)

or no payload at all and it should just be invoked as [%my_ext]:

Ast_pattern.(pstr nil)

You should play with Ast_pattern a bit if you need to express complex patterns as I think it's +the only way to get the hang of it.

Writing a deriver

Registering a deriver is slightly different from registering an extension but in the end it remains +relatively simple and you will still have to provide the actual implementation in the form of an +expand function.

The typical my_ppx_deriver.ml will look like:

open Ppxlib
+
+let str_type_decl_generator =
+  Deriving.Generator.make_no_arg
+    ~attributes
+    expand_str
+
+let sig_type_decl_generator =
+  Deriving.Generator.make_no_arg
+    ~attributes
+    expand_sig
+
+let my_deriver =
+  Deriving.add
+    ~str_type_decl:str_type_decl_generator
+    ~sig_type_decl:sig_type_decl_generator
+    "my_deriver"

Which you'll need to compile with the following library stanza:

(library
+ (public_name my_ppx)
+ (kind ppx_deriver)
+ (libraries ppxlib))

The Deriving.add function is declared as:

val add
+  :  ?str_type_decl:(structure, rec_flag * type_declaration list) Generator.t
+  -> ?str_type_ext :(structure, type_extension                  ) Generator.t
+  -> ?str_exception:(structure, extension_constructor           ) Generator.t
+  -> ?sig_type_decl:(signature, rec_flag * type_declaration list) Generator.t
+  -> ?sig_type_ext :(signature, type_extension                  ) Generator.t
+  -> ?sig_exception:(signature, extension_constructor           ) Generator.t
+  -> ?extension:(loc:Location.t -> path:string -> core_type -> expression)
+  -> string
+  -> t

It takes a mandatory string argument, here "my_deriver", which defines how +user are going to invoke your deriver. In this case we'd need to add a [@@deriving my_deriver] to +a type declaration in a structure or a signature to use it. +Then there's just one optional argument per kind of node to which you can attach a [@@deriving ...] +attribute. type_decl correspond to type = ..., type_ext to type += ... and exception to +exception My_exc of .... +You need to provide generators for the ones you wish your deriver to handle, ppxlib +will make sure users get a compile error if they try to use it elsewhere. +We can ignore the extension as it's just here for compatibility with ppx_deriving.

Now let's take a look at Generator. Its type is defined as ('output_ast, 'input_ast) t where +'input_ast is the type of the node to which the [@@deriving ...] is attached and 'output_ast +the type of the nodes it should produce, ie either a structure or a signature. The type of a +generator depends on the expand function it's built from when you use the smart constructor +make_no_arg meaning the expand function should have type +loc: Location.t -> path: string -> 'input_ast -> 'output_ast. This function is the actual +implementation of your deriver and will generate the list of structure_item or signature_item +from the type declaration.

Compatibility with `ppx_import`

ppx_import is a PPX rewriter that lets you import type +definitions and spares you the need to copy and update them every time they change upstream. The +main reason why you would want to do that is because you need to derive values from those types +using a deriver thus the importance of ensuring your deriving plugin is compatible.

Let's take an example to illustrate how ppx_import is used. I'm using a library called blob +which exposes a type Blob.t. For some reason I need to be able to serialize and deserialize +Blob.t values to JSON. I'd like to use a deriver to do that as I don't want to maintain that code +myself. Imagine Blob.t is defined as:

type t =
+  { value : string
+  ; length : int
+  ; id : int
+  }

Without ppx_import I would define somewhere a serializable_blob type as follows:

type serializable_blob = Blob.t =
+  { value : string
+  ; length : int
+  ; id : int
+  }
+[@@deriving yojson]

That works well especially because the type definition is simple but I don't really care about +having it here, what I really want is just the to_yojson and of_yojson functions. Also now, if +the type definition changes, I have to update it here manually. Maintaining many such imports can be +tedious and duplicates a lot of code unnecessarily.

What I can do instead, thanks to ppx_import is to write it like this:

type serializable_blob = [%import: Blob.t]
+[@@deriving yojson]

which will ultimately be expanded into the above using Blob's definition of the type t.

Now ppx_import works a bit differently from regular PPX rewriters as it needs a bit more information +than just the AST. We don't need to understand how it works but what it means is that if your +deriving plugin is used with ppx_import, it will be called twice:

A first time with ocamldep. This is required to determine the dependencies of a module in terms +of other OCaml modules. PPX-es need to be applied here to find out about dependencies they may +introduce.
A second time before actually compiling the code.

The issue here is that during the ocamldep pass, ppx_import doesn't have the information it +needs to import the type definition yet so it can't copy it and it expands:

type u = [%import A.t]

into:

type u = A.t

Only during the second pass will it actually expand it to the copied type definition.

This may be a concern if your deriving plugin can't apply to abstract types because you will +probably raise an error when encountering one, meaning the first phase will fail and the whole +compilation will fail without giving your rewriter a chance to derive anything from the copied +type definition.

The right way to deal with this is to have different a behaviour in the context of ocamldep. +In this case you can ignore such type declaration or eventually, if you know you are going to +inject new dependencies in your generated code, to create dummy values referencing them and just +behave normally in any other context.

ppxlib versions 0.6.0 and higher allow you to do so through the Deriving.Generator.V2 API +which passes an abstract ctxt value to your expand function instead of a loc and a path. +You can tell whether it is the ocamldep pass from within the expand function like this:

open Ppxlib
+
+let expand ~ctxt input_ast =
+  let omp_config = Expansion_context.Deriver.omp_config ctxt in
+  let is_ocamldep_pass = String.equal "ocamldep" omp_config.Migrate_parsetree.Driver.tool_name in
+  ...

Deriver attributes

You'll have noted the attributes parameter in the examples. It's an optional parameter that lets +you define which attributes your deriver allows the user to attach to various bits of the type, +type extension or exception declaration it is applied to.

ppxlib comes with a Attribute module that lets you to properly declare the attributes you want +to allow and make sure they are properly used: correctly spelled, placed and with the right +payload attached. This is especially useful since attributes are by default ignored by the compiler +meaning without ppxlib's care, plugin users wouldn't get any errors if they misused an attribute +and it might take them a while to figure out they got it wrong and the generated code wasn't +impacted as they hoped. +The Attribute module offers another great feature: Attribute.t values can be used to extract the +attribute payload from an AST node if it is present. That will spare you the need for +inspecting attributes yourself which can prove quite tedious.

Ppxlib.Attribute.t is defined as ('context, 'payload) t where 'context describes to which node +the attribute can be attached and 'payload, the type of its payload. +To build such an attribute you must use Ppxlib.Attribute.declare:

val declare
+  :  string
+  -> 'a Context.t
+  -> (payload, 'b, 'c) Ast_pattern.t
+  -> 'b
+  -> ('a, 'c) t

Let's try to declare the default argument from ppx_deriving_yojson I mentioned earlier.

The first string argument is the attribute name. ppxlib support namespaces for the attributes so +that users can avoid conflicting attributes between various derivers applied to the same type +definitions. For instance here we could use "default". It can prove helpful to use more qualified +name such as "ppx_deriving_yojson.of_yojson.default". That means that our attribute can be used as +[@@default ...], [@@of_yojson.default ...] or [@@ppx_deriving.of_yojson.default ...]. +Now if another deriver uses a [@@default ...], users can apply both derivers and provide different +default values to the different derivers by writing:

type t =
+  { a : int
+  ; b : string [@make.default "abc"] [@of_yojson.default ""]
+  }
+[@@deriving make,of_yojson]

The context argument works very similarly to the one in Extension.declare. Here we want the +attribute to be attached to record field declarations so we'll use +Attribute.Context.label_declaration which has type label_declaration Attribute.Context.t.

The pattern argument is an Ast_pattern.t. Now that we know how to work with those this is pretty +easy. Here we need to accept any expression as a payload since we should be able to apply the +default attribute to any field, regardless of its type and we want to extract that expression from +the payload so we can use it in our deserializer so let's use +Ast_pattern.(single_expr_payload __).

Finally the last 'b argument has the same type as the pattern consumer function. We can use it to +transform what we extracted using the previous Ast_pattern but in this case we just want to +keep the expression as we got it so we'll just use the identity function here.

We end up with the following:

let default_attribute =
+  Attribute.declare
+    "ppx_deriving_yojson.of_yojson.default"
+    Attribute.Context.label_declaration
+    Ast_pattern.(single_expr_payload __)
+    (fun expr -> expr)

and that gives us a (label_declaration, expression) Attribute.t.

You can then use it to collect the attribute payload from a label_declaration:

Attribute.get default_attribute label_decl

which will return Some expr if the attribute was attached to label_decl or None otherwise.

Because of their polymorphic nature, attributes need to be packed, ie to be wrapped with a variant +to hide the type parameter, so if you want to pass it to Generator.make_no_arg you'll have to do +it like this:

let attributes = [Attribute.T default_attribute]

Writing your expand functions

In the two last sections I mentioned expand functions that would contain the actual deriver or +extension implementation but didn't actually said anything about how to write those. It will +depend a lot on the purpose of your PPX rewriter and what you're trying to achieve.

Before writing your PPX you should clearly specify what it should be applied to and what code it +should produce. That will help you declaring the right deriving or extension rewriter and from there +you'll know the type of the expand functions you have to write which should help.

A good way to proceed is to use the dumpast tool to pretty print the AST fragments of both the +input of your expander and the output, ie the code it should generate. To take a concrete example, +say you want to write a deriving plugin that generates an equal function from a type definition. +You can start by running dumpast on the following file:

type some_record =
+  { a : int64
+  ; b : string
+  }
+
+let equal_some_record r r' = Int64.equal r.a r'.a && String.equal r.b r'.b

That will give you the AST representation of a record type definition and the equal function you +want to write so you can figure out how to deconstruct your expander's input to be able to generate +the right output.

ppxlib exposes smart constructors in Ppxlib.Ast_builder.Default to help you build AST fragments +without having to care too much attributes and such fields as well as some convenience constructors +to keep your code concise and readable.

Another convenience tool ppxlib exposes to help you build AST fragments is metaquot. I recently +wrote a bit of documentation about it +here which you +should take a look at but to sum it up metaquot is a PPX extension allowing you to write AST nodes +using the OCaml syntax they describe instead of the AST types.

Handling code locations in a PPX rewriter

When building AST fragments you should keep in mind that you have to set their location. Locations +are part of the AST values that describes the position of the corresponding node in your source +file, including the file name and the line number and offset of both the beginning and the end the +code bit they represent.

Because your code was generated after the file was parsed, it doesn't have a location so you need to +set it yourself. One could think that it doesn't matter and we could use a dummy location but +locations are used by the compiler to properly report errors and that's why a PPX rewriter should care +about how it locates the generated code as it will help the end user to understand whether the error +comes from their code or generated code and how to eventually fix it.

Both Ast_builder and metaquot expect a location. The first explicitly takes it as a labelled +loc argument while the second relies on a loc value being available in the scope. It is +important to set those with care as errors in the generated code doesn't necessarily mean that your +rewriter is bugged. There are valid cases where your rewriter functioned as intended but the generated +code triggers an error. PPX-es often work on the assumption that some values are available in the +scope, if the user doesn't properly provide those it's their responsibility to fix the error. To +help them do so, it is important to properly locate the generated code to guide them as much as +possible.

When writing extensions, using the whole extension point location for the generated code makes +perfect sense as that's where the code will sit. That's fairly easy as this what ppxlib passes +to the expand function through the loc labelled argument. For deriving plugins it's a bit different +as the generated code doesn't replace an existing part of the parsed AST but generate a new one to insert. +Currently ppxlib gives you the loc of the whole type declaration, extension or exception +declaration your deriving plugin is applied to. Ideally it would be nice to be able to locate the +generated code on the plugin name in the deriving attribute payload, ie here:

[@@deriving my_plugin,another_plugin]
+            ^^^^^^^^^

I'm currently working on making that location available to the expand function. In the meantime, +you should choose a convention. I personally locate all the generated code on the +type declaration. Some choose to locate the generated code on the part of the input AST they're +handling when generating it.

Reporting errors to your rewriter users

You won't always be able to handle all the AST nodes passed to your expand functions, either because the +end user misused your rewriter or because there are some cases you simply can't deal with.

In those cases you can report the error to the user with Ppxlib.Location.raise_errorf. It works +similarly to printf and you can build your error message from a format string and extra +arguments. It will then raise an exception which will be caught and reported by the compiler. +A good practice is to prefix the error message with the name of your rewriter to help users understand +what's going on, especially with deriving plugin as they might use several of them on the same type +declaration.

Another point to take care of here is, again, locations. raise_errorf takes a labelled loc +arguments. It is used so that your error is reported as any compiler error. Having good locations in +those error messages is just as important as sending clear error messages. Keep in mind that both +the errors you report yourself or errors coming from your generated code will be highlighted by +merlin so when properly set they make it much easier to work with your PPX rewriter.

Testing your PPX

Just as most pieces of code do, a PPX deserves to be tested and it has become easier over the years to +test rewriters.

I personally tend to write as many unit test as possible for my PPX-es internal libraries. I try to +extract helper functions that can easily be unit-tested but I can't test it all that way. +Testing the ast -> ast functions would be tedious as ppxlib and ocaml-migrate-parsetree +don't provide comparison and pretty printing functions that you can use with alcotest or oUnit. +That means you'd have to import the AST types and derive them on your own. That would make a lot +of boiler plate and even if those functions were exposed, writing such tests would be really +tedious. There's a lot of things to take into account. How are you going to build the input AST values +for instance? If you use metaquot, every node will share the same loc, making it hard to test +that your errors are properly located. If you don't, you will end up with insanely long and +unreadable test code or fixtures. +While that would allow extremely accurate testing for the generated code and errors, it will almost +certainly make your test code unmaintainable, at least given the current tooling.

Don't panic, there is a very good and simple alternative. ppxlib makes it very easy to build a +binary that will parse OCaml code, preprocess the AST with your rewriter and spit it out, formatted as +code again.

You just have to write the following pp.ml:

let () = Ppxlib.Driver.standalone ()

and build the binary with the following dune stanza, assuming your rewriter is called +my_ppx_rewriter:

(executable
+ (name pp)
+ (modules pp)
+ (libraries my_ppx_rewriter ppxlib))

Because we're humans and the OCaml syntax is meant for us to write and read, it makes for much better +test input/output. You can now write your test input in a regular .ml file, use the pp.exe +binary to "apply" your preprocessor to it and compare the output with another .ml file containing +the code you expect it to generate. This kind of test pattern is really well supported by dune +thanks to the diff user action.

I usually have the following files in a rewriter/deriver folder within my test directory:

test/rewriter/
+├── dune
+├── test.expected.ml
+├── pp.ml
+└── test.ml

Where pp.ml is used to produce the rewriter binary, test.ml contains the input OCaml code and +test.expected.ml the result of preprocessing test.ml. The dune file content is generally similar +to this:

(executable
+ (name pp)
+ (modules pp)
+ (libraries my_ppx_rewriter ppxlib))
+
+(rule
+ (targets test.actual.ml)
+ (deps (:pp pp.exe) (:input test.ml))
+ (action (run ./%{pp} -deriving-keep-w32 both --impl %{input} -o %{targets})))
+
+(alias
+ (name runtest)
+ (action (diff test.expected.ml test.actual.ml)))
+
+(test
+  (name test)
+  (modules test)
+  (preprocess (pps my_ppx_rewriter)))

The first stanza is the one I already introduced above and specifies how to build the rewriter binary.

The rule stanza that comes after that indicates to dune how to produce the actual test output by +applying the rewriter binary to test.ml. You probably noticed the -deriving-keep-w32 both CLI +option passed to pp.exe. By default, ppxlib will generate values or add attributes so that your +generated code doesn't trigger a "Unused value" warning. This is useful in real life situation but +here it will just pollute the test output and make it harder to read so we disable that feature.

The following alias stanza is where all the magic happens. Running dune runtest will now +generate test.actual.ml and compare it to test.expected.ml. It will not only do that but show +you how they differ from each other in a diff format. You can then automatically update +test.expected.ml if you're happy with the results by running dune promote.

Finally the last test stanza is there to ensure that the generated code compiles without type +errors.

This makes a very convenient test setup to write your PPX-es TDD style. You can start by writing an +identity PPX, that will just return its input AST as it is. Then you add some OCaml code using your +soon to be PPX in test.ml and run dune runtest --auto-promote to prefill test.expected.ml. +From there you can start implementing your rewriter and run dune runtest to check on your progress +and update the expected result with dune promote. +Going pure TDD by writing the test works but it's tricky cause you'd have to format your code the +same way pp.exe will format the AST. It would be great to be able to specify how to format +the generated test.actual.ml so that this approach would be more viable and the diff more +readable. Being able to use ocamlformat with a very diff friendly configuration would be great +there. pp.exe seems to offer CLI options to change the code style such as -styler but I haven't +had the chance to experiment with those yet.

Now you can test successful rewriting this way but what about errors? There's a lot of value +ensuring you produce the right errors and on the right code location because that's the kind of +things you can get wrong when refactoring your rewriter code or when people try to contribute. +That isn't as likely to happen if your CI yells when you break the error reporting. So how do we do +that?

Well pretty much the exact same way! We write a file with an erroneous invocation of our rewriter, +run pp.exe on it and compare stderr with what we expect it to be. +There are two major differences here. First we want to collect the stderr output of the rewriter +binary instead of using it to generate a file. The second is that we cant write all of our test +cases in a single file since pp.exe will stop at the first error. That means we need one .ml +file per error test case. +Luckily for us, dune offers ways to do both.

For every error test file we will want to add the following stanzas:

(rule
+  (targets test_error.actual)
+  (deps (:pp pp.exe) (:input test_error.ml)) 
+  (action
+    (with-stderr-to
+      %{targets}
+      (bash "./%{pp} -no-color --impl %{input} || true")
+    )
+  )
+)
+
+(alias
+  (name runtest)
+  (action (diff test_error.expected test_error.actual))
+)

but obviously we don't want to do that by hand every time we add a new test case so we're gonna need +a script to generate those stanzas and then include them into our dune file using +(include dune.inc).

To achieve that while keeping things as clean as possible I use the following directory structure:

test/rewriter/
+├── errors
+│   ├── dune
+│   ├── dune.inc
+│   ├── gen_dune_rules.ml
+│   ├── pp.ml
+│   ├── test_some_error.expected
+│   ├── test_some_error.ml
+│   ├── test_some_other_error.expected
+│   └── test_some_other_error.ml
+├── dune
+├── test.expected.ml
+├── pp.ml
+└── test.ml

Compared to our previous setup, we only added the new errors folder. To keep things simple it has +its own pp.ml copy but in the future I'd like to improve it a bit and be able to use the same +pp.exe binary.

The most important files here are gen_dune_rules.ml and dune.inc. The first is just a simple +OCaml script to generate the above stanzas for each test cases in the errors directory. The second +is the file we'll include in the main dune. It's also the file to which we'll write the generated +stanza.

I personally use the following gen_dune_rules.ml:

let output_stanzas filename =
+  let base = Filename.remove_extension filename in
+  Printf.printf
+    {|
+(library
+  (name %s)
+  (modules %s)
+  (preprocess (pps ppx_yojson))
+)
+
+(rule
+  (targets %s.actual)
+  (deps (:pp pp.exe) (:input %s.ml))
+  (action
+    (with-stderr-to
+      %%{targets}
+      (bash "./%%{pp} -no-color --impl %%{input} || true")
+    )
+  )
+)
+
+(alias
+  (name runtest)
+  (action (diff %s.expected %s.actual))
+)
+|}
+    base
+    base
+    base
+    base
+    base
+    base
+
+let is_error_test = function
+  | "pp.ml" -> false
+  | "gen_dune_rules.ml" -> false
+  | filename -> Filename.check_suffix filename ".ml"
+
+let () =
+  Sys.readdir "."
+  |> Array.to_list
+  |> List.sort String.compare
+  |> List.filter is_error_test
+  |> List.iter output_stanzas

Nothing spectacular here, we just build the list of all the .ml files in the directory except +pp.ml and gen_dune_rules.ml itself and then generate the right stanzas for each of them. You'll +note the extra library stanza which I add to get dune to generate the right .merlin so that I +can see the error highlights when I edit the files by hand.

With that we're almost good, add the following to the dune file and you're all set:

(executable
+  (name pp)
+  (modules pp)
+  (libraries
+    ppx_yojson
+    ppxlib
+  )
+)
+
+(include dune.inc)
+
+(executable
+  (name gen_dune_rules)
+  (modules gen_dune_rules)
+)
+
+(rule
+  (targets dune.inc.gen)
+  (deps
+    (:gen gen_dune_rules.exe)
+    (source_tree .)
+  )
+  (action
+    (with-stdout-to
+      %{targets}
+      (run %{gen})
+    ) 
+  )
+)
+
+(alias
+  (name runtest)
+  (action (diff dune.inc dune.inc.gen))
+)

The first stanza is here to specify how to build the rewriter binary, same as before, while the +second stanza just tells dune to include the content of dune.inc within this dune file.

The interesting part comes next. As you can guess the executable stanza builds our little OCaml +script into a .exe. The rule that comes after that specifies how to generate the new stanzas +by running gen_dune_rules and capturing its standard output into a dune.inc.gen file. +The last rule allows you to review the changes to the generated stanza and use promotion to accept +them. Once this is done, the new stanzas will be included to the dune file and the test will be +run for every test cases.

Adding a new test case is then pretty easy, you can simply run:

$ touch test/rewriter/errors/some_explicit_test_case_name.{ml,expected} && dune runtest --auto-promote

That will create the new empty test case and update the dune.inc with the corresponding rules. +From there you can proceed the same way as with the successful rewriting tests, update the .ml, +run dune runtest to take a sneak peek at the output and dune promote once you're satisfied with +the result.

I've been pretty happy with this setup so far although there's room for improvement. It would be +nice to avoid duplicating pp.ml for errors testing. This also involves +quite a bit of boilerplate that I have to copy into all my PPX rewriters repositories every time. +Hopefully dune plugins should help with that and I +can't wait for a first version to be released so that I can write a plugin to make this test +pattern more accessible and easier to set up.

7th MirageOS hack retreat

tarides — 2019-05-06T00:00:00-00:00

Let's talk sun, mint tea and OCaml: Yes, you got it, the MirageOS biennial retreat at Marrakesh!

For the 7th iteration of the retreat, the majority of the Tarides team took part in the trip to the camels country. +This is a report about what we produced and enjoyed while there.

Charles-Edouard Lecat

That's it, my first MirageOS retreat is coming soon, let's jump in the plane and here I come. After a nice cab trip and an uncountable number of similar streets, I'm finally at the Riad which will host me for the next 5 days.

It now begins the time to do what I came for: Code, Eat, Sleep and Repeat

I mostly worked on Colombe, the OCaml implementation of the SMTP protocol for which I developed a simple client. +Except some delayed problems (like the integration of the MIME protocol, the TLS wrapping and some others), the client was working perfectly :) +Implementing it was actually really easy as the core of the SMTP protocol was done by @dinosaure who developed over time a really nice way of implementing this kind of API. And as I spend most of my time at Tarides working on his code, I feel really comfortable with it.

One of the awesome thing about this retreat was the people who came: There was so many interesting people, doing various thing, so each time someone had an interrogation, you could almost be sure that someone could help you in a way or another.

But sadly, as I arrived few days after everyone, and just before the week-end, the time flew away reaaaaaally fast, and I did not have the time to do some major code, but I'm already looking forward to the next retreat which, I am sure, will be even more fruitful and attract a lot of nice OCaml developers.

Until then, I will just dream about the awesome food I ate there ;)

Lucas Pluvinage

Second Mirage retreat for me, and this time I had plans: make a small web game with Mirage hosted by an ESP32 device. I figured out that there was not canonical way to make an HTTP/Websocket server with Mirage and I didn't want to stick to a particular library.

Instead, I took my time to develop mirage-http, an abstraction of HTTP that can either have cohttp or httpaf as a backend. On top of that, I've build mirage-websocket which is therefore an HTTP server-independant implementation of websockets (indeed this has a lot of redundancies with ocaml-websocket but for now it's a proof of concept). While making all this I discussed with @anmonteiro who's the Webservers/protocols expert for Mirage ! However I didn't have the time to build something on top of that, but this is still something that I would like achieve at some point.

I also became the "dune guy" as I'm working on the Mirage/dune integration, and helped some people with their build system struggles.

It was definitely a rich week, I've learnt a lot of things, enjoyed the sun, ate good food and contributed to the Mirage universe !

Jules Aguillon

This was my first retreat. +It was the occasion to meet OCaml developers from all over the world. +The food was great and the weather perfect.

I submitted some PRs to the OCaml compiler !

Hint on type error on int literal PR #2301. +It's adding an hint when using int literals instead of other number literals (eg. 3 instead of 3. or 3L):

Line 2, characters 20-21:
+2 | let _ = Int32.add a 3
+                        ^
+Error: This expression has type int but an expression was expected of type
+          int32
+        Hint: Did you mean `3l'?

Hint on type error on int operators PR #2307. Hint the user when using numerical operators for ints (eg. +) on other kind of numbers (eg. float, int64, etc..). For example:

Line 8, characters 8-9:
+8 | let _ = x + 1.
+            ^
+Error: This expression has type float but an expression was expected of type
+          int
+Line 8, characters 10-11:
+8 | let _ = x + 1.
+              ^
+  Hint: Did you mean to use `+.'?

+
Clean up int literal hint PR #2313. A little cleanup of the 2 previous PRs.
+
+
Hint when the expected type is wrapped in a ref PR #2319. An other PR adding an hint: When the user forgot to use the ! operator on ref values:
+

Line 2, characters 8-9:
+2 | let b = a + 1
+            ^
+Error: This expression has type int ref
+        but an expression was expected of type int
+  Hint: This is a `ref', did you mean `!a'?

The first 3 are merged now.

Gabriel de Perthuis

For this retreat my plan was to do something a little different and work on Solo5.

Wodan, the storage layer I'm working on, +needs two things from its backends which are not commonly implemented:

support for discarding unused blocks (first implemented in mirage-block-unix), and
support for barriers, which are ordering constraints between writes

Solo5 provides relevant mirage backends, which are themselves provided by various +virtualised implementations. Discard was added to most of those, at least those +that were common enough to be easily tested; we just added an "operation not supported" +error code for the other cases.

The virtio implementation was interesting; recent additions to the spec allow discard +support, but few virtual machine managers actually implement that on the backend side. +I tried to integrate with the Chromium OS "crosvm" for that, and had a good time +figuring out how it found the bootloader entry point (turns out the cpu was happily +skipping past invalid instructions to find a slightly misaligned entry point), but +ran out of time to figure out the rest of the integration, which seemed to be more +complex that anticipated. Because of this virtio discard support will be skipped over +for now.

I also visited the souk, which was an interesting experience. +Turns out I'm bad at haggling, but I brought back interesting things anyway.

Conclusion

We'd like to thank Hannes Mehnert who organized this retreat and all the attendees who contributed to make it fruitful and inspiring. +You want to take part in the next MirageOS retreat? Stay tuned here.

Dune 1.9.0

tarides — 2019-04-10T00:00:00-00:00

Tarides is pleased to have contributed to the dune 1.9.0 release which +introduces the concept of library variants. Thanks to this update, +unikernels builds are becoming easier and faster in the MirageOS +universe! This also opens the door for a better cross-compilation +story, which will ease the addition of new MirageOS backends +(trustzone, ESP32, RISC-V, etc.)

This post has also been posted to the +Dune blog. See also the the discuss +forum for more +details.

Dune 1.9.0

Changes include:

Coloring in the watch mode (#1956)
$ dune init command to create or update project boilerplate (#1448)
Allow "." in c_names and cxx_names (#2036)
Experimental Coq support
Support for library variants and default implementations (#1900)

Variants

In dune 1.7.0, the concept of virtual library was introduced: +https://dune.build/blog/virtual-libraries/. This feature allows to +mark some abstract library as virtual, and then have several +implementations for it. These implementations could be for multiple +targets (unix, xen, js), using different algorithms, using C +code or not. However each implementation in a project dependency tree +had to be manually selected. Dune 1.9.0 introduces features for +automatic selection of implementations.

Library variants

Variants is a tagging mechanism to select implementations on the final +linking step. There's not much to add to make your implementation use +variants. For example, you could decide to design a bar_js library +which is the javascript implementation of bar, a virtual +library. All you need to do is specificy a js tag using the +variant option.

(library
+ (name bar_js)
+ (implements bar)
+ (variant js)); <-- variant specification

Now any executable that depend on bar can automatically select the +bar_js library variant using the variants option in the dune file.

(executable
+ (name foo)
+ (libraries bar baz)
+ (variants js)); <-- variants selection

Common variants

Language selection

In your projects you might want to trade off speed for portability:

ocaml: pure OCaml
c: OCaml accelerated by C

JavaScript backend

js: code aiming for a Node backend, using Js_of_ocaml

Mirage backends

The Mirage project (mirage.io) will make +extensive use of this feature in order to select the appropriate +dependencies according to the selected backend.

unix: Unikernels as Unix applications, running on top of mirage-unix
xen: Xen backend, on top of mirage-xen
freestanding: Freestanding backend, on top of mirage-solo5

Default implementation

To facilitate the transition from normal libraries into virtuals ones, +it's possible to specify an implementation that is selected by +default. This default implementation is selected if no implementation +is chosen after variant resolution.

(library
+ (name bar)
+ (virtual_modules hello)
+ (default_implementation bar_unix)); <-- default implementation selection

Selection mechanism

Implementation is done with respect to some priority rules:

manual selection of an implementation overrides everything
after that comes selection by variants
finally unimplemented virtual libraries can select their default implementation

Libraries may depend on specific implementations but this is not +recommended. In this case, several things can happen:

the implementation conflicts with a manually selected implementation: resolution fails.
the implementation overrides variants and default implementations: a cycle check is done and this either resolves or fails.

Conclusion

Variant libraries and default implementations are fully documented +here. This +feature improves the usability of virtual libraries.

This +commit +shows the amount of changes needed to make a virtual library use +variants.

Coq support

Dune now supports building Coq projects. To enable the experimental Coq +extension, add (using coq 0.1) to your dune-project file. Then, +you can use the (coqlib ...) stanza to declare Coq libraries.

A typical dune file for a Coq project will look like:

(include_subdirs qualified) ; Use if your development is based on sub directories
+
+(coqlib
+  (name Equations)                  ; Name of wrapper module
+  (public_name equations.Equations) ; Generate an .install file
+  (synopsis "Equations Plugin")     ; Synopsis
+  (libraries equations.plugin)      ; ML dependencies (for plugins)
+  (modules :standard \ IdDec)       ; modules to build
+  (flags -w -notation-override))    ; coqc flags

See the documentation of the +extension for more +details.

Credits

This release also contains many other changes and bug fixes that can +be found on the discuss +announce.

Special thanks to dune maintainers and contributors for this release: +@rgrinberg, +@emillon, +@shonfeder +and @ejgallego!

Release of OCamlFormat 0.9

tarides — 2019-03-29T00:00:00-00:00

We are pleased to announce the release of OCamlFormat (available on opam). +There have been numerous changes since the last release, +so here is a comprehensive list of the new features and breaking changes to help the transition from OCamlFormat 0.8.

Additional dependencies

OCamlFormat now requires:

ocaml >= 4.06 (up from 4.04.1)
dune >= 1.1.1
octavius >= 1.2.0
uutf

OCamlFormat_Reason now requires:

ocaml >= 4.06
dune >= 1.1.1
ocaml-migrate-parsetree >= 1.0.10 (up from 1.0.6)
octavius >= 1.2.0
uutf
reason >= 3.2.0 (up from 1.13.4)

New preset profiles

The ocamlformat profile aims to take advantage of the strengths of a parsetree-based auto-formatter, +and to limit the consequences of the weaknesses imposed by the current implementation. +This is a style which optimizes for what the formatter can do best, rather than to match the style of any existing code. +General guidelines that have directed the design include:

Legibility, in the sense of making it as hard as possible for quick visual parsing to give the wrong interpretation, +is of highest priority;
Whenever possible the high-level structure of the code should be obvious by looking only at the left margin, +in particular, it should not be necessary to visually jump from left to right hunting for critical keywords, tokens, etc;
All else equal compact code is preferred as reading without scrolling is easier, +so indentation or white space is avoided unless it helps legibility;
Attention has been given to making some syntactic gotchas visually obvious.

ocamlformat is the new default profile.

The conventional profile aims to be as familiar and "conventional" appearing as the available options allow.

The default profile is ocamlformat with break-cases=fit. +default is deprecated and will be removed in version 0.10.

OCamlFormat diff tool

ocamlformat-diff is a tool that uses OCamlFormat to apply the same formatting to compared OCaml files, +so that the formatting differences between the two files are not displayed. +Note that ocamlformat-diff comes in a separate opam package and is not included in the ocamlformat package.

The file comparison is then performed by any diff backend.

The options' documentation is available through ocamlformat-diff --help.

The option --diff allows you to configure the diff command that is used to compare the formatted files. +The default value is the vanilla diff, but you can also use patdiff or any other similar comparison tool.

ocamlformat-diff can be integrated with git diff, +as explained in the online documentation.

Formatting docstrings

Previously, the docstrings (** This is a docstring *) could only be formatted like regular comments, +a new option --parse-docstrings has been added so that docstrings can be nicely formatted.

Here is a small example:

(** {1 Printers and escapes used by Cmdliner module} *)
+
+val subst_vars : subst:(string -> string option) -> Buffer.t -> string -> string
+(** [subst b ~subst s], using [b], substitutes in [s] variables of the form
+    "$(doc)" by their [subst] definition. This leaves escapes and markup
+    directives $(markup,...) intact.
+    @raise Invalid_argument in case of illegal syntax. *)

Note that this option is disabled by default and you have to set it manually by adding --parse-docstrings to your command line +or parse-docstrings=true to your .ocamlformat file. +If you get the following error message:

Error: Formatting of (** ... *) is unstable (e.g. parses as a list or not depending on the margin), please tighten up this comment in the source or disable the formatting using the option --no-parse-docstrings.

It means the original docstring cannot be formatted (e.g. because it does not comply with the odoc syntax) +and you have to edit it or disable the formatting of docstrings.

Of course if you think your docstring complies with the odoc syntax and there might be a bug in OCamlFormat, +feel free to file an issue on github.

Print the configuration

The new --print-config flag prints the configuration determined by the environment variable, +the configuration files, preset profiles and command line. Attributes are not considered.

It provides the full list of options with the values they are set to, and the source of this value. +For example ocamlformat --print-config prints:

profile=ocamlformat (file .ocamlformat:1)
+quiet=false (profile ocamlformat (file .ocamlformat:1))
+max-iters=10 (profile ocamlformat (file .ocamlformat:1))
+comment-check=true (profile ocamlformat (file .ocamlformat:1))
+wrap-fun-args=true (profile ocamlformat (file .ocamlformat:1))
+wrap-comments=true (file .ocamlformat:5)
+type-decl=compact (profile ocamlformat (file .ocamlformat:1))
+space-around-collection-expressions=false (profile ocamlformat (file .ocamlformat:1))
+single-case=compact (profile ocamlformat (file .ocamlformat:1))
+sequence-style=separator (profile ocamlformat (file .ocamlformat:1))
+parse-docstrings=true (file .ocamlformat:4)
+parens-tuple-patterns=multi-line-only (profile ocamlformat (file .ocamlformat:1))
+parens-tuple=always (profile ocamlformat (file .ocamlformat:1))
+parens-ite=false (profile ocamlformat (file .ocamlformat:1))
+ocp-indent-compat=false (profile ocamlformat (file .ocamlformat:1))
+module-item-spacing=sparse (profile ocamlformat (file .ocamlformat:1))
+margin=77 (file .ocamlformat:3)
+let-open=preserve (profile ocamlformat (file .ocamlformat:1))
+let-binding-spacing=compact (profile ocamlformat (file .ocamlformat:1))
+let-and=compact (profile ocamlformat (file .ocamlformat:1))
+leading-nested-match-parens=false (profile ocamlformat (file .ocamlformat:1))
+infix-precedence=indent (profile ocamlformat (file .ocamlformat:1))
+indicate-nested-or-patterns=space (profile ocamlformat (file .ocamlformat:1))
+indicate-multiline-delimiters=true (profile ocamlformat (file .ocamlformat:1))
+if-then-else=compact (profile ocamlformat (file .ocamlformat:1))
+field-space=tight (profile ocamlformat (file .ocamlformat:1))
+extension-sugar=preserve (profile ocamlformat (file .ocamlformat:1))
+escape-strings=preserve (profile ocamlformat (file .ocamlformat:1))
+escape-chars=preserve (profile ocamlformat (file .ocamlformat:1))
+doc-comments-tag-only=default (profile ocamlformat (file .ocamlformat:1))
+doc-comments-padding=2 (profile ocamlformat (file .ocamlformat:1))
+doc-comments=after (profile ocamlformat (file .ocamlformat:1))
+disable=false (profile ocamlformat (file .ocamlformat:1))
+cases-exp-indent=4 (profile ocamlformat (file .ocamlformat:1))
+break-struct=force (profile ocamlformat (file .ocamlformat:1))
+break-string-literals=wrap (profile ocamlformat (file .ocamlformat:1))
+break-sequences=false (profile ocamlformat (file .ocamlformat:1))
+break-separators=before (profile ocamlformat (file .ocamlformat:1))
+break-infix-before-func=true (profile ocamlformat (file .ocamlformat:1))
+break-infix=wrap (profile ocamlformat (file .ocamlformat:1))
+break-fun-decl=wrap (profile ocamlformat (file .ocamlformat:1))
+break-collection-expressions=fit-or-vertical (profile ocamlformat (file .ocamlformat:1))
+break-cases=fit (file .ocamlformat:2)

If many input files are specified, only print the configuration for the first file. +If no input file is specified, print the configuration for the root directory if specified, +or for the current working directory otherwise.

Parentheses around if-then-else branches

A new option parens-ite has been added to decide whether to use parentheses +around if-then-else branches that spread across multiple lines.

If this option is set, the following function:

let rec loop count a =
+  if count >= self#len
+  then a
+  else
+    let a' = f cur#get count a in
+    cur#incr ();
+    loop (count + 1) a'

will be formatted as:

let rec loop count a =
+  if count >= self#len
+  then a
+  else (
+    let a' = f cur#get count a in
+    cur#incr ();
+    loop (count + 1) a' )

Parentheses around tuple patterns

A new option parens-tuple-patterns has been added, that mimics parens-tuple but only applies to patterns, +whereas parens-tuples only applies to expressions. +parens-tuple-patterns=multi-line-only mode will try to skip parentheses for single-line tuple patterns, +this is the default value. +parens-tuple-patterns=always always uses parentheses around tuples patterns.

For example:

(* with parens-tuple-patterns=always *)
+let (a, b) = (1, 2)
+
+(* with parens-tuple-patterns=multi-line-only *)
+let a, b = (1, 2)

Single-case pattern-matching expressions

The new option single-case defines the style of pattern-matching expressions with only a single case. +single-case=compact will try to format a single case on a single line, this is the default value. +single-case=sparse will always break the line before a single case.

For example:

(* with single-case=compact *)
+try some_irrelevant_expression
+with Undefined_recursive_module _ -> true
+
+(* with single-case=sparse *)
+try some_irrelevant_expression
+with
+| Undefined_recursive_module _ -> true

Space around collection expressions

The new option space-around-collection-expressions decides whether to add a space +inside the delimiters of collection expressions (lists, arrays, records).

For example:

(* by default *)
+type wkind = {f : 'a. 'a tag -> 'a kind}
+let l = ["Nil", TCnoarg Thd; "Cons", TCarg (Ttl Thd, tcons)]
+
+(* with space-around-collection-expressions *)
+type wkind = { f : 'a. 'a tag -> 'a kind }
+let l = [ "Nil", TCnoarg Thd; "Cons", TCarg (Ttl Thd, tcons) ]

Break separators

The new option break-separators decides whether to break before or after separators such as ; in list or record expressions, +* in tuples or -> in arrow types. +break-separators=before breaks the expressions before the separator, this is the default value. +break-separators=after breaks the expressions after the separator. +break-separators=after-and-docked breaks the expressions after the separator and docks the brackets for records.

For example:

(* with break-separators=before *)
+type t =
+  { foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo
+  ; fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo }
+
+(* with break-separators=after *)
+type t =
+  { foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo;
+    fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo }
+
+(* with break-separators=after-and-docked *)
+type t = {
+  foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo;
+  fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo
+}

Not breaking before bind/map operators

The new option break-infix-before-func decides whether to break infix operators +whose right arguments are anonymous functions specially. +This option is set by default, if you disable it with --no-break-infix-before-func, +it will not break before the operator so that the first line of the function appears docked at the end of line after the operator.

For example:

(* by default *)
+f x
+>>= fun y ->
+g y
+>>= fun () ->
+f x >>= fun y -> g y >>= fun () -> f x >>= fun y -> g y >>= fun () -> y ()
+
+(* with break-infix-before-func = false *)
+f x >>= fun y ->
+g y >>= fun () ->
+f x >>= fun y -> g y >>= fun () -> f x >>= fun y -> g y >>= fun () -> y ()

Break toplevel cases

There is a new value for the break-cases option: toplevel, +that forces top-level cases (i.e. not nested or-patterns) to break across lines, +otherwise breaks naturally at the margin.

For example:

let f =
+  let g = function
+    | H when x y <> k -> 2
+    | T | P | U -> 3
+  in
+  fun x g t h y u ->
+    match x with
+    | E -> 4
+    | Z | P | M -> (
+      match y with
+      | O -> 5
+      | P when h x -> (
+          function
+          | A -> 6 ) )

Number of spaces before docstrings

The new option doc-comments-padding controls how many spaces are printed before doc comments in type declarations. +The default value is 2.

For example:

(* with doc-comments-padding = 2 *)
+type t = {a: int  (** a *); b: int  (** b *)}
+
+(* with doc-comments-padding = 1 *)
+type t = {a: int (** a *); b: int (** b *)}

Ignore files

An .ocamlformat-ignore file specifies files that OCamlFormat should ignore. +Each line in an .ocamlformat-ignore file specifies a filename relative to the directory containing the .ocamlformat-ignore file. +Lines starting with # are ignored and can be used as comments.

Here is an example of such .ocamlformat-ignore file:

#This is a comment
+dir2/ignore_1.ml

Tag-only docstrings

The new option doc-comments-tag-only controls the position of doc comments only containing tags. +doc-comments-tag-only=default means no special treatment is done, this is the default value. +doc-comments-tag-only=fit puts doc comments on the same line if it fits.

For example:

(* with doc-comments-tag-only = default *)
+
+(** @deprecated  *)
+open Module
+
+(* with doc-comments-tag-only = fit *)
+
+open Module (** @deprecated  *)

Fit or vertical mode for if-then-else

There is a new value for the option if-then-else: fit-or-vertical. +fit-or-vertical vertically breaks all branches if they do not fit on a single line. +Compared to the compact (default) value, it breaks all branches if at least one of them does not fit on a single line.

For example:

(* with if-then-else = compact *)
+let _ =
+  if foo then
+    let a = 1 in
+    let b = 2 in
+    a + b
+  else if foo then 12
+  else 0
+
+(* with if-then-else = fit-or-vertical *)
+let _ =
+  if foo then
+    let a = 1 in
+    let b = 2 in
+    a + b
+  else if foo then
+    12
+  else
+    0

Check mode

A new --check flag has been added. +It checks whether the input files already are formatted. +This flag is mutually exclusive with --inplace and --output. +It returns 0 if the input files are indeed already formatted, or 1 otherwise.

Break function declarations

The new option break-fun-decl controls the style for function declarations and types. +break-fun-decl=wrap breaks only if necessary, this is the default value. +break-fun-decl=fit-or-vertical vertically breaks arguments if they do not fit on a single line. +break-fun-decl=smart is like fit-or-vertical but try to fit arguments on their line if they fit. +The wrap-fun-args option now only controls the style for function calls, and no more for function declarations.

For example:

(* with break-fun-decl = wrap *)
+let ffffffffffffffffffff aaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
+    cccccccccccccccccccccc =
+  g
+
+(* with break-fun-decl = fit-or-vertical *)
+let ffffffffffffffffffff
+    aaaaaaaaaaaaaaaaaaaaaa
+    bbbbbbbbbbbbbbbbbbbbbb
+    cccccccccccccccccccccc =
+  g
+
+(* with break-fun-decl = smart *)
+let ffffffffffffffffffff
+    aaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb cccccccccccccccccccccc =
+  g

Disable configuration in files and attributes

Two new options have been added so that .ocamlformat configuration files and attributes in OCaml files do not change the +configuration. +These options can be useful if you use some preset profile +and you do not want attributes and .ocamlformat files to interfere with your preset configuration. +--disable-conf-attrs disables the configuration in attributes, +and --disable-conf-files disables .ocamlformat configuration files.

Preserve module items spacing

There is a new value for the option module-item-spacing: preserve, +that will not leave open lines between one-liners of similar sorts unless there is an open line in the input.

For example the line breaks are preserved in the following code:

let cmos_rtc_seconds = 0x00
+let cmos_rtc_seconds_alarm = 0x01
+let cmos_rtc_minutes = 0x02
+
+let x = o
+
+let log_other = 0x000001
+let log_cpu = 0x000002
+let log_fpu = 0x000004

Breaking changes

When --disable-outside-detected-project is set, disable ocamlformat when no .ocamlformat file is found.
Files are not parsed when ocamlformat is disabled.
Disallow - with other input files.
The wrap-fun-args option now only controls the style for function calls, and no more for function declarations.
The default profile is now named ocamlformat.
The deprecated syntax for .ocamlformat files: option value is no more supported anymore and you should use the option = value syntax instead.

Miscellaneous bugfixes

Preserve shebang (e.g. #!/usr/bin/env ocaml) at the beginning of a file.
Improve the formatting when ocp-indent-compat is set.
UTF8 characters are now correctly printed in comments.
Add parentheses around a constrained any-pattern (e.g. let (_ : int) = x1).
Emacs: the temporary buffer is now killed.
Emacs: add the keybinding in tuareg's map instead of merlin's.
Lots of improvements on the comments, docstrings, attributes formatting.
Lots of improvements on the formatting of modules.
Lots of improvements in the Reason support.
Do not rely on the file-system to format sources.
The --debug mode is more user-friendly.

Credits

This release also contains many other changes and bug fixes that we cannot detail here.

Special thanks to our maintainers and contributors for this release: Jules Aguillon, Mathieu Barbin, Josh Berdine, Jérémie Dimino, Hugo Heuzard, Ludwig Pacifici, Guillaume Petiot, Nathan Rebours and Louis Roché.

If you wish to get involved with OCamlFormat development or file an issue, +please read the contributing guide, +any contribution is welcomed.

Release of Base64

tarides — 2019-02-08T00:00:00-00:00

MirageOS is a library operating system written from the ground up in OCaml. +It has an impossible and incredibly huge goal to re-implement all of the +world! Looking back at the work accomplished by the MirageOS team, it appears that's +what happened for several years. Re-implementing the entire stack, in particular +the lower layers that we often take for granted, requires a great attention to +detail. While it may seem reasonably easy to implement a given RFC, a huge +amount of work is often hidden under the surface.

In this article, we will explain the development process we went through, as we +updated a small part of the MirageOS stack: the library ocaml-base64. It's a +suitable example as the library is small (few hundreds lines of code), but it +needs ongoing development to ensure good quality and to be able to trust it for +higher level libraries (like mrmime).

Updating the library was instigated by a problem I ran into with the existing +base64 implementation while working on the e-mail stack. Indeed, we got some +errors when we tried to compute an encoded-word according to the RFC +2047. So after several years of not being touched, we decided to +update ocaml-base64.

The Critique of Pure Reason

The first problem

We started by attempting to use ocaml-base64 on some examples extracted from +actual e-mails, and we quickly ran into cases where the library failed. This +highlighted that reality is much more complex than you can imagine from reading +an RFC. In this situation, what do you do: try to implement a best-effort +strategy and continue parsing? Or stick to the letter of the RFC and fail? In +the context of e-mails, which has accumulated a lot of baggage over time, you +cannot get around implementing a best-effort strategy.

The particular error we were seeing was a Not_found exception when decoding an +encoded-word. This exception appeared because the implementation relied on +String.contains, and the input contained a character which was not part of the +base64 alphabet.

This was the first reason why we thought it necessary to rewrite ocaml-base64. +Of course, we could just catch the exception and continue the initial +computation, but then another reason appeared.

The second problem

As @clecat and I reviewed RFC 2045, we noticed the +following requirement:

+
The encoded output stream must be represented in lines of no more than 76 +characters each.
+
See RFC 2045, section 6.8
+

Pretty specific, but general to e-mails, we should never have more than 78 +characters per line according to RFC 822, nor more than 998 characters +according to RFC 2822.

Having a decoder that abided RFC 2045 more closely, including the requirement +above, further spurred us to implement a new decoder.

As part of the new implementation, we decided to implement tests and fuzzers to +ensure correctness. This also had the benefit, that we could run the fuzzer on +the existing codebase. When fuzzing an encoder/decoder pair, an excellent check +is whether the following isomorphism holds:

let iso0 input = assert (decode (encode input) = input)
+let iso1 input = assert (encode (decode input) = input)

However, at this point @hannesm ran into another error (see +#20).

The third problem

We started to review the nocrypto implementation of base64, which +respects our requirements. We had some concerns about the performance of the +implementation though, so we decided to see if we would get a performance +regression by switching to this implementation.

A quick benchmark based on random input revealed the opposite, however! +nocrypto's implementation was faster than ocaml-base64:

ocaml-base64's implementation on bytes (length: 5000): 466 272.34ns
+nocrypto's implementation on bytes (length: 5000): 137 406.04ns

Based on all these observations, we thought there was sufficient reason to +reconsider the ocaml-base64 implementation. It's also worth mentioning that +the last real release (excluding dune/jbuilder/topkg updates) is from Dec. +24 2014. So, it's pretty old code and the OCaml eco-system has improved a lot +since 2014.

Implementation & review

We started integrating the nocrypto implementation. Of course, implementing +RFC 4648 is not as easy as just reading examples and trying to do +something which works. The devil is in the detail.

@hannesm and @cfcs decided to do a big review of expected behavior +according to the RFC, and another about implementation and security issues.

Canonicalization

The biggest problem about RFC 4648 is regarding canonical inputs. Indeed, there +are cases where two different inputs are associated with the same value:

let a = Base64.decode "Zm9vCg==" ;;
+- : string = "foo\n"
+let b = Base64.decode "Zm9vCh==" ;;
+- : string = "foo\n"

This is mostly because the base64 format encodes the input 6 bits at a time. The +result is that 4 base64 encoded bytes are equal to 3 decoded bytes (6 * 4 = 8 * 3). Because of this, 2 base64 encoded bytes provide 1 byte plus 4 bits. What do +we need to do with these 4 bits? Nothing.

That's why the last character in our example can be something else than g. g +is the canonical byte to indicate using the 2 bits afterward the 6 bits +delivered by C (and make a byte - 8 bits). But h can be used where we just +need 2 bits at the end.

Due to this behavior, the check used for fuzzing changes: from a canonical +input, we should check isomorphism.

Invalid character

As mentioned above ("The first problem"), how should invalid characters be +handled? This happens when decoding a byte which is not a part of the base64 +alphabet. In the old version, ocaml-base64 would simply leak a Not_found +exception from String.contains.

The MirageOS team has taken a stance on exceptions, which is +to "use exceptions for exceptional conditions" - invalid input is hardly one of +those. This is to avoid any exception leaks, as it can be really hard to track +the origin of an exception in a unikernel. Because of this, several packages +have been updated to return a result type instead, and we wanted the new +implementation to follow suit.

On the other hand, exceptions can be useful when considered as a more +constrained form of assembly jump. Of course, they break the control flow, but +from a performance point of view, it's interesting to use this trick:

exception Found
+
+let contains str chr =
+  let idx = ref 0 in
+  let len = String.length str in
+  try while !idx < len
+      do if String.unsafe_get str !idx = chr then raise Found ; incr idx done ;
+      None
+  with Found -> Some !idx

This kind of code for example is ~20% faster than String.contains.

As such, exceptions can be a useful tool for performance optimizations, but we +need to be extra careful not to expose them to the users of the library. This +code needs to be hidden behind a fancy functional interface. With this in mind, +we should assert that our decode function never leaks an exception. We'll +describe how we've adressed this problem later.

Special cases

RFC 4648 has some detailed cases and while we would sometimes like to work in a +perfect world where we will never need to deal with such errors, from our +experience, we cannot imagine what the end-user will do to formats, protocols +and such.

Even though the RFC has detailed examples, we have to read between lines to know +special cases and how to deal with them.

@hannesm noticed one of these cases, where padding (= sign at the end of +input) is not mandatory:

+
The pad character "=" is typically percent-encoded when used in an URI [9], +but if the data length is known implicitly, this can be avoided by skipping +the padding; see section 3.2.
+
See RFC 4648, section 5
+

That mostly means that the following kind of input can be valid:

let a = Base64.decode ~pad:false "Zm9vCg"
+- : string = "foo\n"

It's only valid in a specific context though: when length is known implicitly. +Only the caller of decode can determine whether the length is implicitly known +such that padding can be omitted. To that end, we've added a new optional +argument ?pad to the function Base64.decode.

Allocation, `sub`, `?off` and `?len`

Xavier Leroy has described the garbage collector in the following way:

+
You see, the Caml garbage collector is like a god from ancient mythology: +mighty, but very irritable. If you mess with it, it'll make you suffer in +surprising ways.
+

That's probably why my experience with improving the allocation policy of +(ocaml-git)ocaml-git was quite a nightmare. Allowing the user to control +allocation is important for efficiency, and we wanted to ocaml-base64 to be a +good citizen.

At the beginning, ocaml-base64 had a very simple API:

val decode : string -> string
+val encode : string -> string

This API forces allocations in two ways.

Firstly, if the caller needs to encode a part of a string, this part needs to be +extracted, e.g. using String.sub, which will allocate a new string. To avoid +this, two new optional arguments have been added to encode: ?off and ?len, +which specifies the substring to encode. Here's an example:

(* We want to encode the part 'foo' without prefix or suffix *)
+
+(* Old API -- forces allocation *)
+Base64.encode (String.sub "prefix foo suffix" 7 3) ;;
+- : string = "Zm9v"
+
+(* New API -- avoids allocation *)
+Base64.encode ~off:7 ~len:3 "prefix foo suffix" ;;
+- : string = "Zm9v"

Secondly, a new string is allocated to hold the resulting string. We can +calculate a bound on the length of this string in the following manner:

let (//) x y =
+  if y < 1 then raise Division_by_zero ;
+  if x > 0 then 1 + ((x - 1) / y) else 0
+
+let encode input =
+  let res = Bytes.create (String.length input // 3 * 4) in
+  ...
+
+let decode input =
+  let res = Bytes.create (String.length input // 4 * 3) in
+  ...

Unfortunately we cannot know the exact length of the result prior to computing +it. This forces a call to String.sub at the end of the computation to return a +string of the correct length. This means we have two allocations rather than +one. To avoid the additional allocation, [@avsm][avsm] proposed to provide a new +type sub = string * int * int. This lets the user call String.sub if +required (and allocate a new string), or use simply use the returned sub for +_blit_ting to another buffer or similar.

Fuzz everything!

There's a strong trend of fuzzing libraries for MirageOS, which is quite easy +thanks to the brilliant work by @yomimono and @stedolan! +The integrated fuzzing in OCaml builds on American fuzzy lop, which is +very smart about discarding paths of execution that have already been tested and +generating unseen inputs which break your assumptions. My first experience with +fuzzing was with the library decompress, and I was impressed by +precise error it found about a name clash.

Earlier in this article, I listed some properties we wanted to check for +ocaml-base64:

The functions encode and decode should be be isomorphic taking +canonicalization into account:

let iso0 input =
+  match Base64.decode ~pad:false input with
+  | Error _ -> fail ()
+  | Ok result0 ->
+    let result1 = Base64.encode_exn result0 in
+    match Base64.decode ~pad:true result1 with
+    | Error _ -> fail ()
+    | Ok result2 -> check_eq result0 result2
+
+let iso1 input =
+  let result = Base64.encode_exn input in
+  match Base64.decode ~pad:true result0 with
+  | Error _ -> fail ()
+  | Ok result1 ->
+    let result2 = Base64.encode_exn result1 in
+    check_eq result0 result2

The function decode should never raise an exception, but rather return a +result type:

let no_exn input =
+  try ignore @@ Base64.decode input with _ -> fail ()

And finally, we should randomize ?off and ?len arguments to ensure that we +don't get an Out_of_bounds exception when accessing input.

Just because we've applied fuzzing to the new implementation for a long time, it +doesn't mean that the code is completely infallible. People can use our library +in an unimaginable way (and it's mostly what happens in the real world) and get +an unknowable error.

But, with the fuzzer, we've managed to test some properties across a very wide +range of input instead of unit testing with random (or not so random) inputs +from our brains. This development process allows fixing the semantics of +implementations (even if it's not a formal definition of semantics), but +it's better than nothing or outdated documentation.

Conclusion

Based on our recent update to ocaml-base64, this blog post explains our +development process as go about rewriting the world to MirageOS, one bit at a +time. There's an important point to be made though:

ocaml-base64 is a small project. Currently, the implementation is about 250 +lines of code. So it's a really small project. But as stated in the +introduction, we are fortunate enough to push the restart button of the computer +world - yes, we want to make a new operating system.

That's a massive task, and we shouldn't make it any harder on ourselves than +necessary. As such, we need to justify any step, any line of code, and why we +decided to spend our time on any change (why we decided to re-implement git +for example). So before committing any time to projects, we try to do a deep +analysis of the problem, get feedback from others, and find a consensus between +what we already know, what we want and what we should have (in the case of +ocaml-base64, @hannesm did a look on the PHP implementation and the Go +implementation).

Indeed, this is a hard question which nobody can answer perfectly in isolation. +So, the story of this update to ocaml-base64 is an invitation for you to enter +the arcanas of the computer world through MirageOS :) ! Don't be afraid!

How configurator reads C constants

tarides — 2019-01-03T00:00:00-00:00

Dune comes with a library to query OS-specific information, called configurator. +It is able to evaluate C expressions and turn them into OCaml value. +Surprisingly, it even works when compiling for a different architecture. How can +it do that?

MirageOS, towards a smaller and safer OS

tarides — 2018-12-06T00:00:00-00:00

Presentation about MirageOS in Lambda World Cadìz on October 26th

ocaml-git 2.0

tarides — 2018-10-19T00:00:00-00:00

I'm very happy to announce a new major release of ocaml-git (2.0). +This release is a 2-year effort to get a revamped +streaming API offering a full control over memory +allocation. This new version also adds production-ready implementations of +the wire protocol: git push and git pull now work very reliably +using the raw Git and smart HTTP protocol (SSH support will come +soon). git gc is also implemented, and all of the basic bricks are +now available to create Git servers. MirageOS support is available +out-of-the-box.

Two years ago, we decided to rewrite ocaml-git and split it into +standalone libraries. More details about these new libraries are also +given below.

But first, let's focus on ocaml-git's new design. The primary goal was +to fix memory consumption issues that our users noticed with the previous version, +and to make git push work reliably. We also took care about +not breaking the API too much, to ease the transition for current users.

Controlled allocations

There is a big difference in the way ocaml-git and git +are designed: git is a short-lived command-line tool which does not +care that much about allocation policies, whereas we wanted to build a +library that can be linked with long-lived Git client and/or server +applications. We had to make some (performance) compromises to support +that use-case, at the benefit of tighter allocation policies — and hence +more predictable memory consumption patterns. +Other Git libraries such as libgit2 +also have to deal with similar concerns.

In order to keep a tight control on the allocated memory, we decided to +use decompress instead of +camlzip. decompress allows the users to provide their own buffer +instead of allocating dynamically. This allowed us to keep a better +control on memory consumption. See below for more details on decompress.

We also used angstrom and +encore to provide a streaming interface +to encode and decode Git objects. The streaming API is currently hidden +to the end-user, but it helped us a lot to build abstraction and, again, on +managing the allocation policy of the library.

Complete PACK file support (including GC)

In order to find the right abstraction for manipulating pack files in +a long-lived application, we experimented with +various +prototypes. We haven't found the +right abstractions just yet, but we believe the PACK format could be useful +to store any kind of data in the future (and not especially Git objects).

We implemented git gc by following the same heuristics as +Git +to compress pack files and +we produce something similar in size — decompress has a good ratio about +compression — and we are using duff, our own implementation of xdiff, the +binary diff algorithm used by Git (more details on duff below). +We also had to re-implement the streaming algorithm to reconstruct idx files on +the fly, when receiving pack file on the network.

One notable feature of our compression algorithms is they work without +the assumption that the underlying system implements POSIX: hence, +they can work fully in-memory, in a browser using web storage or +inside a MirageOS unikernel with wodan.

Production-ready push and pull

We re-implemented and abstracted the Git Smart protocol, and used that +abstraction to make git push and git pull work over HTTP. By +default we provide a cohttp +implementation but users can use their own — for instance based on +httpaf. +As proof-of-concept, the initial +pull-request of ocaml-git 2.0 was +created using this new implementation; moreover, we wrote a +prototype of a Git client compiled with js_of_ocaml, which was able +to run git pull over HTTP inside a browser!

Finally, that implementation will allow MirageOS unikernels to synchronize their +internal state with external Git stores (hosted for instance on GitHub) +using push/pull mechanisms. We also expect to release a server-side implementation +of the smart HTTP protocol, so that the state of any unikernel can be inspected +via git pull. Stay tuned for more updates on that topic!

Standalone Dependencies

Below you can find the details of the new stable releases of libraries that are +used by ocaml-git 2.0.

`optint` and `checkseum`

In some parts of ocaml-git, we need to compute a Circular +Redundancy Check value. It is 32-bit integer value. optint provides +an abstraction of it but structurally uses an unboxed integer or a +boxed int32 value depending on target (32 bit or 64 bit architecture).

checkseum relies on optint and provides 3 implementations of CRC:

Adler32 (used by zlib format)
CRC32 (used by gzip format and git)
CRC32-C (used by wodan)

checkseum uses the linking trick: this means that users of the +library program against an abstract API (only the cmi is provided); +at link-time, users have to select which implementation to use: +checkseum.c (the C implementation) or checkseum.ocaml (the OCaml +implementation). The process is currently a bit cumbersome but upcoming +dune release will make that process much more transparent to the users.

`encore` (/angkor/)

In git, we work with Git objects (tree, blob or +commit). These objects are encoded in a specific format. Then, +the hash of these objects are computed from the encoded +result to get a unique identifier. For example, the hash of your last commit is: +sha1(encode(commit)).

A common operation in git is to decode Git objects from an encoded +representation of them (especially in .git/objects/* as a loose +file) and restore them in another part of your Git repository (like in a +PACK file or on the command-line).

Hence, we need to ensure that encoding is always deterministic, and +that decoding an encoded Git object is always the identity, e.g. there is +an isomorphism between the decoder and the encoder.

let decoder <.> encoder : value -> value = id
+let encoder <.> decoder : string -> string = id

encore is a library in which you +can describe a format (like Git format) and from it, we can derive a +streaming decoder and encoder that are isomorphic by +construction.

`duff`

duff is a pure implementation in +OCaml of the xdiff algorithm. +Git has an optimized representation of your Git repository. It's a +PACK file. This format uses a binary diff algorithm called xdiff +to compress binary data. xdiff takes a source A and a target B and try +to find common sub-strings between A and B.

This is done by a Rabin's fingerprint of the source A applied to the +target B. The fingerprint can then be used to produce a lightweight +representation of B in terms of sub-strings of A.

duff implements this algorithm (with additional Git's constraints, +regarding the size of the sliding windows) in OCaml. It provides a +small binary xduff that complies with the format of Git without the zlib +layer.

$ xduff diff source target > target.xduff
+$ xduff patch source < target.xduff > target.new
+$ diff target target.new
+$ echo $?
+0

`decompress`

decompress +is a pure implementation in OCaml of zlib and +rfc1951. You can compress and decompress data flows and, obviously, +Git does this compression in loose files and PACK files.

It provides a non-blocking interface and is easily usable in a server +context. Indeed, the implementation never allocates and only relies on +what the user provides (window, input and output buffer). Then, the +distribution provides an easy example of how to use decompress:

val inflate: ?level:int -> string -> string
+val deflate: string -> string

`digestif`

digestif is a toolbox providing +many implementations of hash algorithms such as:

MD5
SHA1
SHA224
SHA256
SHA384
SHA512
BLAKE2B
BLAKE2S
RIPEMD160

Like checkseum, digestif uses the linking trick too: from a +shared interface, it provides 2 implementations, in C (digestif.c) +and OCaml (digestif.ocaml).

Regarding Git, we use the SHA1 implementation and we are ready to +migrate ocaml-git to BLAKE2{B,S} as the Git core team expects - and, +in the OCaml world, it is just a functor application with +another implementation.

`eqaf`

Some applications require that secret values are compared in constant +time. Functions like String.equal do not have this property, so we +have decided to provide a small package — eqaf — +providing a constant-time equal function. +digestif uses it to check equality of hashes — it also exposes +unsafe_compare if you don't care about timing attacks in your application.

Of course, the biggest work on this package is not about the +implementation of the equal function but a way to check the +constant-time assumption on this function. Using this, we did a +benchmark on Linux, +Windows and Mac to check it.

An interesting fact is that after various experiments, we replaced the +initial implementation in C (extracted from OpenBSD's timingsafe_memcmp) with an OCaml +implementation behaving in a much more predictable way on all the +tested platforms.

Conclusion

The upcoming version 2.0 of Irmin is using ocaml-git +to create small applications that push and pull their state +to GitHub. +We think that Git offers a very nice model to persist data for distributed +applications and we hope that more people will use ocaml-git to experiment +and manipulate application data in Git. Please +send us your feedback!

OCamlFormat 0.8

tarides — 2018-10-17T00:00:00-00:00

We are proud to announce the release of OCamlFormat 0.8 (available on opam). To ease the transition from the previous 0.7 release here are some highlights of the new features of this release. The full changelog is available on the project repository.

Precedence of options

In the previous version you could override command line options with .ocamlformat files configuration. 0.8 fixed this so that the OCamlFormat configuration is first established by reading .ocamlformat and .ocp-indent files:

margin = 77
+wrap-comments = true

By default, these files in current and all ancestor directories for each input file are used, as well as the global configuration defined in $XDG_CONFIG_HOME/ocamlformat. The global $XDG_CONFIG_HOME/ocamlformat configuration has the lowest priority, then the closer the directory is to the processed file, the higher the priority. In each directory, both .ocamlformat and .ocp-indent files are read, with .ocamlformat files having the higher priority.

For now ocp-indent options support is very partial and is expected to be extended in the future.

Then the parameters can be overriden with the OCAMLFORMAT environment variable:

OCAMLFORMAT=field-space=tight,type-decl=compact

and finally the parameters can be overriden again with the command lines parameters.

Reading input from stdin

It is now possible to read the input from stdin instead of OCaml files. The following command invokes OCamlFormat that reads its input from the pipe:

echo "let f x = x + 1" | ocamlformat --name a.ml -

The - on the command line indicates that ocamlformat should read from stdin instead of expecting input files. It is then necessary to use the --name option to designate the input (a.ml here).

Preset profiles

Preset profiles allow you to have a consistent configuration without needing to tune every option.

Preset profiles set all options, overriding lower priority configuration. A preset profile can be set using the --profile (or -p) option. You can pass the option --profile=<name> on the command line or add profile = <name> in an .ocamlformat configuration file.

The available profiles are:

default sets each option to its default value
compact sets options for a generally compact code style
sparse sets options for a generally sparse code style
janestreet is the profile used at JaneStreet

To get a better feel of it, here is the formatting of the mk_callback function from the OCaml compiler with the compact profile:

let mk_callback rest name desc = function
+  | None -> nothing
+  | Some f -> (
+      fun () ->
+        match rest with
+        | [] -> f name None
+        | (hidden, _) :: _ -> f name (Some (desc, hidden)) )

then the same function formatted with the sparse profile:

let mk_callback rest name desc = function
+  | None ->
+      nothing
+  | Some f ->
+      fun () ->
+        ( match rest with
+        | [] ->
+            f name None
+        | (hidden, _) :: _ ->
+            f name (Some (desc, hidden)) )

Project root

The project root of an input file is taken to be the nearest ancestor directory that contains a .git or .hg or dune-project file. +If the new option --disable-outside-detected-project is set, .ocamlformat configuration files are not read outside of the current project. If no configuration file is found, formatting is disabled.

A new option, --root allows to specify the root directory for a project. If specified, OCamlFormat only takes into account .ocamlformat configuration files inside the root directory and its subdirectories.

Credits

This release also contains many other changes and bug fixes that we cannot detail here. Check out the full changelog.

Special thanks to our maintainers and contributors for this release: David Allsopp, Josh Berdine, Hugo Heuzard, Brandon Kase, Anil Madhavapeddy and Guillaume Petiot.

OCaml Workshop 2018

tarides — 2018-09-27T00:00:00-00:00

The OCaml Users and Developers Workshop brings together industrial +users of OCaml with academics and hackers who are working on extending +the language, type system and tools. OCaml 2018 was held on September +27th, 2018 in St. Louis, Missouri, USA, colocated with ICFP 2018.

Check Tarides' talks: RFCs, all the way down! and The OCaml Platform 1.0. +

Dune 1.2.0

tarides — 2018-09-06T00:00:00-00:00

After a tiny but important patch release as 1.1.1, the dune team is thrilled to +announce the release of dune 1.2.0! Here are some highlights of the new +features in that version. The full list of changes can be found in the dune +repository.

Watch mode

When developing, it is common to edit a file, run a build, read the error +message, and fix the error. Since this is a very tight loop and developers are +doing this hundreds or thousands times a day, it is crucial to have the +quickest feedback possible.

dune build and dune runtest now accept a -w +flag that will +watch the filesystem for changes, and trigger a new build.

Better error messages

Inspired by the great work done in +Elm and +bucklescript, +dune now displays the relevant file in error messages.

 % cat dune
+(executable
+ (name my_program)
+ (librarys cmdliner)
+)
+ % dune build
+File "dune", line 3, characters 2-10:
+3 |  (librarys cmdliner)
+      ^^^^^^^^
+Error: Unknown field librarys
+Hint: did you mean libraries?

Many messages have also been improved, in particular to help users switching +from the jbuild format to the dune +format.

dune unstable-fmt

Are you confused about how to format S-expressions? You are not alone. +That is why we are gradually introducing a formatter for dune files. It can +transform a valid but ugly dune into one that is consistently formatted.

 % cat dune
+(executable( name ls) (libraries cmdliner)
+(preprocess (pps ppx_deriving.std)))
+ % dune unstable-fmt dune
+(executable
+ (name ls)
+ (libraries cmdliner)
+ (preprocess
+  (pps ppx_deriving.std)
+ )
+)

This feature is not ready yet for the end user (hence the unstable part), +and in particular the concrete syntax is not stable yet. +But having it already in the code base will make it possible to build useful +integrations with dune itself (to automatically reformat all dune files in a +project, for example) and common editors, so that they format dune files on +save.

First class support of findlib plugins

It is now easy to support findlib plugins by just adding the findlib.dynload +library dependency. Then you can use Fl_dynload module in your code which +will automatically do the right thing. A complete example can be found in the +dune manual.

Promote only certain files

The dune promote command now accept a list of files. This is useful to +promote just the file that is opened in a text editor for example. Some emacs +bindings are provided to do this, which works particularly well with +inline expectation tests.

Deprecation message for (wrapped) modes

By default, libraries are (wrapped true), which means that they expose a +single OCaml module (source files are exposed as submodules of this main +module). This is usually desired as it makes link-time name collisions less +likely. However, a lot of libraries are using (wrapped false) (expose all +source files as modules) to keep compatibility.

It can be challenging to transition from (wrapped false) to (wrapped true) +because it breaks compatibility. That is why we have added (wrapped (transition "message")) which will generate wrapped modules but keep unwrapped +modules with a deprecation message to help coordinate the change.

Credits

Special thanks to our contributors for this release: @aantron, @anuragsoni, +@bobot, @ddickstein, @dra27, @drjdn, @hongchangwu, @khady, @kodek16, +@prometheansacrifice and @ryyppy.

Station F

tarides — 2018-07-17T00:00:00-00:00

We are thrilled to have been accepted into the Founders Progam's 3rd +batch at Station F! Station F is +"the only startup campus gathering a whole entrepreneurial ecosystem +under one roof" and is providing 3000+ desks and 26 international +startup programs. Our Paris offices are now located in that incredible +place, close to "métro Chevaleret" (Paris 13).

If you are in Paris, drop us an email to visit +our beautiful campus!

OCaml Users in Paris (OUPS)

tarides — 2018-05-23T00:00:00-00:00

Thomas Gazagnaire gave a presentation on MirageOS to the +OCaml meetup in Paris.

Check the slides +for more details.

MirageOS + Tezos funding

tarides — 2018-05-23T00:00:00-00:00

We are excited to announce that the Tezos Foundation +will trust Tarides to package Tezos nodes as MirageOS unikernel, which will help participants +establish nodes on the Tezos network in a more efficient and secure manner.

Irmin usability enhancements

tarides — 2018-05-18T00:00:00-00:00

Zach Shipko is working on improving the UI/UX for Irmin. +He is looking for feedback +to make Irmin more accessible to potential users and clean up the rough edges for existing users.

Invited lecture at ENS

tarides — 2018-05-17T00:00:00-00:00

Thomas Gazagnaire gave an invited lecture at +the computer science department of ENS, +in Paris. This was part of the system and network L3 course.

Check the slides (in english) +and the exercices (in french).

HotPOST'18

tarides — 2018-04-16T00:00:00-00:00

Anil Madhavapeddy and Gemma Gordon presented our new operating system for +connected buildings: OSMOSE +to HotPOST’18. OSMOSE is based on +MirageOS and Irmin and we hope to explore that area more in the coming months!

An Architecture for Interspatial Communication

tarides — 2018-02-14T00:00:00-00:00

Position paper on +“An Architecture for Interspatial Communication” +accepted to HotPOST’18.