RFC: Mozzi 2.0 part 1: Single compilation unit

[tfry-git]

This is a write-up of something I have been pondering for quite some time, now: One of the larger problems with Mozzi is that in order to change something in the config (and the need to do so is not uncommon at all), you have to modify config files in the library itself. To spell out some of the problems with that:

• It's really unfriendly to new users
• Dependent projects (using Mozzi) that require a certain config have to pass the same problem to their users
• It's really annoying when working on several projects in parallel
• It prevents us from having automated testing coverage on non-default configuration options such as stereo output

That problem has been felt, before, so why didn't we fix it, yet?

• Switching to run-time configuration is not an option (or at least not for everything), for performance and code-size concerns
• We cannot currently allow users to e.g. customize AUDIO_RATE at the top of their sketch

Why can't we have the latter? Compilation units. For a very rough summary, every .cpp-file is compiled, separately, and only in a final stage everything is linked together. One such .cpp-file is the user's sketch. Another, for example is MozziGuts.cpp. Both also include several common .h-files, but importantly, where they make use of defines such as AUDIO_RATE, this has to match in all compilation units. If AUDIO_RATE was defined at the top of the user-sketch, the MozziGuts.cpp compilation unit would not "see" that define. The current way to make things work, is thus to have both MozziGuts.cpp and the user sketch include a common configuration file. Due to the way the arduino build system works (I won't go into details, mostly, because I don't have them at the top of my head, but I did investigate, before), this common configuration file can only reside in the library folder itself. -> The situation we are in.

So what's the alternative? Putting everything into a single compilation unit. Note this does not mean a single code file, but rather than having MozziGuts.cpp include MozziGuts.h, we'd have MozziGuts.h include (a renamed) MozziGuts.hpp (and the same for all other .cpp files inside Mozzi). In effect, starting with the user sketch as the first file the preprocessor will lump all #included code files together into one big file, and compile that in a single go. Thus a #define near the top of the user sketch will affect all Mozzi code files, too.

What are the drawbacks of that plan?

• Compilation times may go up a bit, because any change in the user sketch will require a recompilation of everything inside Mozzi. I don't really see this as an issue, though. Mozzi code size is not that huge in the first place. The majority of compilation times "heavy" platforms such as ESP32 is spent in other libraries that will not be affected.
• It may not be technically trivial: Importantly we need to make sure, local variables and function names in the Mozzi .cpp files do not clash with user code, and not with locals in other .cpp files inside Mozzi, either. All of that should be fixable, somehow, possibly using namespaces or some-such. However, it may take some time to get there. This, in turn, may mean it will be difficult to address other, unrelated fixes and additions to Mozzi, without ending up in a terrible mess.
• Simple sketches should continue to work, unchanged, but, importantly, sketches consisting of more than one .cpp-file, themselves, will need extra effort. This is actually my biggest concern, but in order to ever finish this write-up at all, I'll detail my thoughts on that in a separate comment (hopefully, within a few days at most).
• Users who have already done some customizations to their Mozzi config will have to re-do those in their sketches. In other words, this is not going to be fully backwards compatible.

What are some side-benefits?

• In theory the compiler should be able to perform more optimizations. Whether that works out remains to be seen.

What else should we worry about?

• Since we're already breaking existing custom configurations, we should seize the opportunity and do some cleanup, there. Importantly, what I have in mind is to use the same configuration defines across all platforms (although not all of them will be supported, everywhere). I'll post details on that, separately, as well. It's not directly related to the plan outlined, here, but should be addressed at the same time, to keep the phase of transition short, and should thus also be mostly planned and agreed on, before we actually start doing anything.

So what will this mean for user sketches? Sketches using the default config will remain 100% unchanged. Others might look like:

#include <MozziCustomConfig.h>
#define AUDIO_RATE 32768
#define AUDIO_CHANNELS 2
#define AUDIO_CHANNEL_1_PIN 5

#include <MozziGuts.h>  // this line and everything below unchanged

---

So, as indicated, I'll write up some further details over the coming days, but of course I'd also like to hear your thoughts on the general plan. Also, what else has always annoyed you about Mozzi, but wasn't quite fixable in a backwards compatible way? Now might be the time to plan that, too.

[tfry-git]

Alright, so what about user sketches which have several compilation units (i.e. several .cpp files) themselves? As long as only one of those files includes Mozzi headers, that's not actually a problem. If several do, we run into two issues:

1. We'll end up with several implementations (one for each unit) of the Mozzi functions. The linker will complain about that.
2. We again need to make sure that each unit uses identical configuration parameters (unless they are at default values, of course; see above).

To address this, the recommendation will be to create a common header file inside the user sketch, containing the configuration:

#include <MozziCommonConfig.h>
#define AUDIO_RATE 32768
// etc.

One source file would then start like this:

#include "MyCustomConfig.h"  // i.e. the file, above
#include <MozziGuts.h>

All other source files would start with:

#include "MyCustomConfig.h"
#define MOZZI_HEADERS_ONLY  // this signifies that Mozzi headers shall only include their declaration, not their implementation
#include <MozziGuts.h>

I guess you can see why this is my biggest worry, here. On the other hand, I believe that most sketches will only ever contain one .cpp file, and users creating multi-compilation-unit sketches will be more experienced, too, on average.

[tomcombriat]

Hi,

Indeed, getting custom configs to work without having to edit the source would be a great improvement! I personally run Mozzi on a lot of different configs, also for testing purposes and, even thought I have no problem changing the config, I inevitably run into errors by forgetting to do so. I remember reading that it is a limitation from the Arduino IDE, that has been asked to be removed to work in a similar way than PlatformIO (I think) by other library dev, with no success. I do not think I have the in-depth knowledge of the compilation process to be of much help in conceptualizing this but I am happy to help where I can, in particular in testing.

I have one question though (might be very naive, bare with me ;)), what happens to the #include "MyCustomConfig.h" for the default config? Basically, if there is no custom configuration, like for some of the examples the compiler will complain that this file does not exist, is there a way to branch that out in that case? Maybe with ifincluded or so?

As per other big changes, nothing comes to mind at the moment that would break backward compatibility. I had the thinking for some time to improve further the fixmath provided by Mozzi, which I think can be a bit hard to grasp, for instance, when multiplying one Q8n8 with a Q16n16, what should be the shift to end up with a correct Q16n16? Indeed, that could can be inferred by:
(QAnB) = (QCnD * QEnF) >> (F+D-B)
but I wonder if that could be templated in some way to add some fixmath operations that gives straight the result in an expected fixmath type. If only we could template the strings of the typenames ;)…

[sensorium]

I'm mostly a spectator these days, but am enjoying the show...

Being able to config within sketches would be great, and your plans sound well considered, Thomas.

And Tom, I agree that the fixed maths is really inelegant at the moment, unfamiliar and confusing for most people! It's in lots of the example sketches and I imagine it's a barrier for many users. Hard to do much synthesis on AVRs without it, though. Any improvements it would be welcome.

Well, since you ask:

what else has always annoyed you about Mozzi

I have a list! Feel free to ignore these, they are just personal gripes...

• fixed math, mentioned already
• naming conventions, for files/modules, function names, object oriented vs functional approaches. Maybe this is all OK but I didn't necessarily resolve things clearly at the start and was improvising as I went. I'm sure it could be done more rationally, but I also guess it could become a bigger job than is worthwhile for a hobby/enthusiast project. (It's not something I would enjoy doing!)
• blocking and non-blocking code needs more explaining/demonstration. Writing interfaces for i2c modules is tedious and there needs to be a place for written interfaces to be collected. Sometimes non-blocking synthesis fails when certain modules send large varying sized messages.
• Mozzi needs a proper line-segment control object which adsr could be based on as well as being able to be made into more flexible multi-segment lines
• a polyphonic voice allocation class would be useful
• Not sure if this is relevant here, but the Mozzi web page is hard to maintain for a lazy, disorganised person. It relies on Jekyll https://jekyllrb.com, and has at times been broken and hard to fix locally, as well as using a collection of local scripts for generating the online examples, testing, recording and uploading all the audio. It used to be streamlined, but a decade later I avoid it like the plague. That just sounds like a complaint...
• there are definitely other things that are annoying about Mozzi, but that's what comes to mind for now.

So, no actual practical help at the moment here, but just my 2 cents...

[tfry-git]

Sorry for the long delay (I'm afraid, it won't be the last time):

@tomcombriat

what happens to the #include "MyCustomConfig.h" for the default config?

That #include is at the level of the user sketch, not Mozzi itself (as an aside, one day I had the idea, that the library could include a config file from inside the sketch directory, but alas, this turned out to be positively impossible). Therefore, if it is not needed, one can simply omit it.

The idea is simply to document this as recommended practice, in case that more than one cpp file exists in the user sketch. We can, but do not have to follow the same in our examples (which are all single-cpp-file, as far as I am aware).

I think I also have some ideas on how to detect (most) cases, where users might have an inconsistent config across two cpp files, and show a clear error message in that case.

@sensorium
All of that makes sense. In this particular context, however, I think "naming conventions" is the only thing that would really need to be addressed for a 2.0, as everything else has no risk of breaking backwards compatibility. Arguably, of course the line-segment control may fall into the same category, as it might involve small API changes in ADSR and/or other classes.

---

Anyway, I'm thrilled to see you're both generally positive towards this slightly megalomaniac plan. At the same time it's beginning to dawn on me that I should try to actually start implementing it :-o. To make the overall work appear less daunting, here's an additional suggestion for getting (some) concepts to work, in a step-by-step fashion:

Would you think an extra folder "playground" (inside the master branch) could be a good idea? It would be excluded from API-docs, and it would carry a Readme that warns against using the code for real, but might be an area, where we can more easily share concepts under development (such as fixedMaths++, but I'm also thinking about some macros that should help with config checking, for instance), without actually having to merge all of Mozzi 2.0 in a single huge MR.

[tomcombriat]

That #include is at the level of the user sketch, not Mozzi itself

Indeed, I must have skipped the part where you said:

the recommendation will be to create a common header file inside the user sketch

I think (but might not be aware of all the implications) that a branch might be more suited than a playground. Of course having a separated branch might lead to one huge MR but at the same time moving this folder to the root will lead to one huge commit. With separate branch(es?) we can consider merging things which do not break compatibility on the go, and also between devel branches (for instance fixMath2 into Mozzi2) and might allow us to be a bit more modular in the approach like branching out from Mozzi2 for testing purposes or new and independent features. It might also give us "clean" sketches for testing instead of importing everything as "#include<playgroud/whatever.h" (I might be completely wrong here…).
At the same time it makes these new features less accessible for anyone else who want to test or use nightly built features, and the partial merges from devel to master might give us quite a few headaches so after writing this I am not really sure…

In all cases, I think there is a good set of potential improvements of I am really looking forward to them and to help where I can! Not having to change mozzi_config.h between different setups would really be something…

[tfry-git]

Good points. Let's make it so.

[tfry-git]

As a small update, I'll have to admit that one drawback I failed to foresee for the "single compilation unit", is that add our internal #defines (and we have a lot of those), are now potentially spilling out, i.e. once MozziGuts has been included, internal stuff like "LOOP_YIELD" would now be "available" in user code, and possibly cause clashes, there. Contrary to regular functions and variables, these cannot simply be confined into a namespace, either, as they exist purely in the preprocessor.

I have two-and-a-half ideas to mitigate this:

1. Quite a few of these can probably be converted to C-level consts (or constexprs or inline functions), with - theoretically - no impact on the assembly produced (a theory, we should test, carefully, though).
2. All others should be prefixed with MOZZI__ (two underscores), in order to avoid name clashes, and document that they are meant for internal use.
3. It would also be possible to undef them after use, but I'm a little concerned that this would make the code yet harder to read.