Problem with app chunking

[hitecSmartHome]

Little back story.

I had a monorepo where I stored my frontend and my esp32 codes.
It worked fine but the frontend started to grow big.

I split my app into two repos. Frontend and backend.
I configured svelteesp32 to build the header file into the other directory.
Worked fine for two build. After the third build I noticed that PsychicHTTP started to give me garbage files.

I decided to split my app into more chunks with Vite, hoping it will solve the garbage output.
Vite produced some file names with @ in it which breaks the compilation.

 [assets\@tanstack-on1rpc6_.js]                 ✓ gzip used (37440 -> 11103 = 30%)      
 [assets\@remix-run-DXtzjFCd.js]                ✓ gzip used (47135 -> 16613 = 35%)      
 [assets\@radix-ui-BmDFjYc9.js]                 ✓ gzip used (115780 -> 33311 = 29%)     
 [assets\@floating-ui-D-N9agw5.js]              ✓ gzip used (21359 -> 8262 = 39%)       
 [assets\@babel-CSh3sF0B.js]                    ✓ gzip used (1412 -> 767 = 54%)

This produced some of these defines #define SVELTEESP32_FILE_ASSETS_@BABEL_CSH3SF0B_JS. Totally broke the header file.
I have replaced every occourence of this _@ to this _. App compiled again but the frontend is still garbage.
I have started to get some files correctly but one or two always broken and the js can't run in the browser. Sometimes it is the html file which is broken ( index.html ) but most of the time it is some js file.

Most of the files are OK but

Here is the Vite chunking code

vite.config.ts

import path from "path"
import react from "@vitejs/plugin-react-swc"
import { defineConfig } from "vite"

export default defineConfig({
  plugins: [react()],
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "./src"),
    },
  },
  build: {
    rollupOptions: {
      output: {
        manualChunks(id) {
          if (id.includes('node_modules')) {
            return id.toString().split('node_modules/')[1].split('/')[0].toString();
          }
        }
      }
    }
  }
})

If I remove this chunking and let Vite handle, I will have only a few files ( without @ sign in them )

[SvelteESP32] Generate code for psychic2 engine
Collecting source files

Translation to header file
 [index.html]                           ✓ gzip used (1552 -> 614 = 40%)
 [manifest\site.webmanifest]            x gzip unused (too small) (878 -> 329 = 37%)    
 [manifest\safari-pinned-tab.svg.gz]    x gzip unused (too small) (545 -> 568 = 104%)   
 [manifest\mstile-150x150.png]          x gzip unused (1642 -> 1555 = 95%)
 [manifest\maskable_icon.png]           ✓ gzip used (3568 -> 2960 = 83%)
 [manifest\favicon.ico]                 ✓ gzip used (7406 -> 1625 = 22%)
 [manifest\favicon-32x32.png]           x gzip unused (too small) (876 -> 896 = 102%)   
 [manifest\favicon-16x16.png]           x gzip unused (too small) (713 -> 728 = 102%)   
 [manifest\browserconfig.xml]           x gzip unused (too small) (273 -> 182 = 67%)    
 [manifest\apple-touch-icon.png]        x gzip unused (1862 -> 1826 = 98%)
 [manifest\android-chrome-512x512.png]  ✓ gzip used (16737 -> 10144 = 61%)
 [manifest\android-chrome-256x256.png]  x gzip unused (too small) (786 -> 712 = 91%)    
 [manifest\android-chrome-192x192.png]  x gzip unused (1701 -> 1641 = 96%)
 [manifest\admin.webmanifest]           x gzip unused (too small) (822 -> 274 = 33%)    
 [images\littleLogo.png]                x gzip unused (too small) (564 -> 559 = 99%)    
 [assets\index-UShPNtFn.css]            ✓ gzip used (47691 -> 8888 = 19%)
 [assets\index-BJ1uG5xx.js]             ✓ gzip used (1085935 -> 332228 = 31%)

17 files, 1146kB original size, 357kB gzip size
../HsH_Core/src/webServer/frontend.h 1294kB size

But the web app won't run regardless. I get garbage on the browser.
Yes I have cleared the cache on the browser. Even did a full clean in PlatformIO.

Will investigate if it is my fault or the PsychicHTTP library fault but something is broken.

About the file naming problem.

Should the user deal with it or should it be the library's business?
In theory if we want manual chunks, we can manipulate the names here.

build: {
    rollupOptions: {
      output: {
        manualChunks(id) {
          if (id.includes('node_modules')) {
            return id.toString().split('node_modules/')[1].split('/')[0].toString(); // Return whatever you want
          }
        }
      }
    }
  }

[hitecSmartHome]

I have discovered chunkSplitPlugin meanwhile. Added it to my project and now it does not output garbage.
Seems I need more testing. Something gets broken somewhere sometimes :D

import path from "path"
import react from "@vitejs/plugin-react-swc"
import { chunkSplitPlugin } from 'vite-plugin-chunk-split';
import { defineConfig } from "vite"

export default defineConfig({
  plugins: [
    react(),
    chunkSplitPlugin()
  ],
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "./src"),
    },
  }
})

[BCsabaEngine]

Thank you for sharing your project details, it is very useful for developing!

We will make a change to handle properly all file names and you can use @ or any other in files names, without breaking the result cpp code.

[BCsabaEngine]

Apart from the file name problem, I think it is important to say that you need to minimize the file count and file size. Some vite plugin can reduce files and bundle it to one JS file. You cannot get around that you must use lot of image file (because web standard requires it), but one JS and one CSS is the target scope.

[BCsabaEngine]

Some expreienced web delveloper say, the chunking makes the webpage loading faster. It is important when SEO matters, but in microcontroller world and home automation topic the SEO does not matter, the compact fileset does matters.

@hitecSmartHome May I ask what kind of solutions you have, which are React codes running on ESP32?

[hitecSmartHome]

Well. SEO doesn't matter thats true. Unfortunately there is no sweet spot for an mcu. If I let few chunks they will be big. If I make smaller chunks, then there will be a lot of requests. Neither is good for a microcontroller. :D Maybe I can fine tune so the react components will load / page and not everything at once.

Iam making a home automation app.

[BCsabaEngine]

Release: https://github.com/BCsabaEngine/svelteesp32/releases/tag/v1.5.1

[mathieucarbou]

Did you try to set FILE_CHUNK_SIZE to 1K ?

In Psychic, FILE_CHUNK_SIZE is set to 8K by default, which is enormous considering that Psychic supports concurrent requests. Also, reading a stream (blocking i/o operation) could halt some other processing. I/O are the slowest operations.

My point of view is that for MCU it is better to limit concurrency but ensure a fast request / response rate than reducing speed in favour of concurrent request. The MCU only has 2 cores and is not preemptive so tasks have to work in a cooperative mode and thus execute really fast small pieces of unit of work each time to be efficient. So even if 8 tasks are spawned, and / or do a lot of long I/O, they will just fight against themselves to get some CPU.

How it works in ESPAsyncWS, when sending chunk, it calls through the AsyncClient tcp_sndbuf which gives back the available buffer for tcp send: the app can write, then as to wait for client reception, and can continue to fill the buffer, etc. So there is no heap buffer like FILE_CHUNK_SIZE.

[hitecSmartHome]

Will try that. Thank you very much for the clarification.