Building a builder: go deeper

In a previous article I created a couple of builders. To be honest, just one + a set of plugins.

Dall-E, just to get your attention

Here, I’ll break this down into parts and reorganize it to create a useful product, rather than a hastily put-together solution.

TL;DR;

GitHub repository: https://github.com/lexey111/insane-nano-builder

Live page: https://lexey111.github.io/insane-nano-builder/

Bonus: https://github.com/lexey111/insane-nano-preact and live page: extended with Preact.

High level diagrams

Building the frontend, or bundling… the difference is subtle but significant, I believe: while bundler just packs everything together, builder does the transformations and processing. For example, Typescript compilation, transpilation, favicon provisioning, dependencies resolving, tree shaking, SASS, etc. Not that bundlers don’t do that too, but it’s a question of scale. Sorry, just got distracted.

This is the diagram of how builder works… well, not how it works inside with all that virtual FS, queues, dependency trees, conveyors, loaders, transformers, plugins, events. No.

This is an entry-level article on how to quickly create a builder for your specific needs without spending too many resources or investing in learning yet another new Webpack.

Well, here it is:

Plain build

Start-build-stop.

For serve, I mean watch mode + web development server, it’s a bit more complicated:

Build serve

Seems a bit messy? Well, it is.

There are three submodules: build, web server, source code watcher.

• build does, err, the build. The results of build are build artefacts — files in dist folder in our case, or build errors — error messages reported by the builder. Errors can be related to any part of the source code — JS, TS, stylesheet, asset, anything that interferes with a successful build.
• web server is a module that serves localhost:3000 and displays the content of dist folder. It is also responsible for delivering notifications, error messages, reloading the page when it is rebuilt and reestablishing the connection.
• watcher is an infinite loop inside that some code detects any change in src folder and initiates a rebuild.

We have, technically, 3 infinite loops:

1. Source code watch → rebuild.
2. ./dist watch (build results) → inform browser → serve files. There’s a fork here: we need to somehow report both success and error.
3. Wait forCtrl-C for exit. This one we got automatically from Node in the top level loop but worth to be mentioned.

Implementation details

Builder

I am using ESBuild as a build core. What’s more; it does all the rest — watching and serving.

Of course, it does this, um, sub-optimally.

I can agree with the ESBuild author here — anything not related to the build itself should be maintained by separate tools. There are a couple of major issues: recovery and debouncing.

import esbuild from 'esbuild'
import htmlPlugin from '@chialab/esbuild-plugin-html'
import {sassPlugin} from 'esbuild-sass-plugin'

const ctx = await esbuild.context({
    logLevel: 'debug',
    entryPoints: [
        'src/index.html',
        'src/dashboard.html'
    ],
    assetNames: 'assets/[name]-[hash]',
    chunkNames: '[ext]/[name]-[hash]',
    outdir: 'dist',
    bundle: true,
    banner: {
        js: 'new EventSource("/esbuild").addEventListener(' +
            '"change", () => setTimeout(() => location.reload(), 1000));'
    },
    plugins: [
        htmlPlugin(),
        sassPlugin()
    ],
})

console.log('✅ Build complete... running watch.\n')

await ctx.watch()

await ctx.serve({
    servedir: 'dist',
    port: 3000,
})

What can we expect from 30 lines of code? It works.

Let’s start with decomposition. I will put everything related to build into build folder.

We’ll just write it down:

❯ du -hs ./node_modules
122M ./node_modules

Thus, let’s install dependencies for separate web server, watcher and typescript (for type check) accordingly:

bun i -D browser-sync
bun i -D chokidar
bun i -D typescript

❯ du -hs ./node_modules
141M ./node_modules

Uh-oh. 141M. With a planned JS output of less than 100Kb. Uh-oh.

Interesting fact: if you don’t install typescript, the size will be 151 MB. Peer deps can sometimes be mysterious…

Type check

I already have JS/TS linting, which is very useful for the IDE. It will catch most of the errors, but not TS related ones.

A function call with an invalid parameter type is highlighted in the IDE…

But the build is passed despite the error

Oops. Clearly this is because TS is just a transpiler and can recommend, but not enforce…

Rather surprisingly, the type checking for TS is integrated with ESBuild rather poorly.

I’ve mentioned https://www.npmjs.com/package/@jgoz/esbuild-plugin-typecheck plugin before, but it does not work correctly, especially with serve mode at the time of writing (Jun’23, 2024).

There are lot of tips on the internet about how to check typescript files. The most robust one — I need to run tsc --noEmit in isolated mode ("isolatedModules": true in tsconfig.json ).

Let’s implement utility function:

// build/utils/ts-check.js

export async function spawnTSC(inheritStdio = false) {
    const tsc = spawnSync(
        'bun tsc --noEmit',
        [],
        {
            cwd: './',
            encoding: 'utf-8',
            stdio: inheritStdio? 'inherit' : 'pipe',
            shell: true,
        }
    )

    if (inheritStdio) {
        return tsc.status
    }

    return {
        messages: tsc.stdout,
        error: tsc.error,
    }
}

A few points to note. First, inheritStdio parameter: I pass that because in build mode I need just a result, a status code. For serve mode I need the console errors to be redirected to a variable for further processing and display.

Second, if you are using NPM, you should replace bun tsc... to npm run tsc... and the corresponding entry in package.json:

  "scripts": {
    ...
    "tsc": "tsc --noEmit",
    ...
  },

Nice, let’s improve build script:

import esbuild from 'esbuild'
import {buildConfig} from './build.config.js'
import {spawnTSC} from './utils/ts-check.js'  // <-- this

console.time('Build time')
console.log('\n[ Starting type check... ]')

try {
    const typeCheck = await spawnTSC(true)   // <-- and this

    if (typeCheck !== 0) {
        process.exit(typeCheck)
    }

    console.log('\n✅  Check complete.\n')

    console.log('[ Building... ]')
    await esbuild.build({
        ...buildConfig,
        metafile: true,
        minify: true,
        minifyWhitespace: true,
        minifySyntax: true,
        minifyIdentifiers: true,
        sourcemap: true,
    })

    console.log('\n✅  Build complete.')

} catch (error) {
    console.log('\n⛔  Build failed.')
    console.error(error)
}

console.timeEnd('Build time')

Try run:

Type check works!

Browser-sync

https://browsersync.io/ is a bit overkill. It allows you to synchronise multiple browsers, and just serving one is kind of a humiliation. But I’ve tried others, such as live-server, and I like this one the best. It has lots of customisation options, is well documented, convenient to use and works great.

Chokidar

https://github.com/paulmillr/chokidar it’s kind of an industry standard. of course, with actual NodeJS even on Linux it could be just a few lines of code, but it adds nothing to node_modules size.

So, that is the build-dev script, esbuild-serve:

import bs from 'browser-sync'
import esbuild from 'esbuild'
import {buildConfig, DEBOUNCE_BUILD_TIME} from './build.config.js'
import {fillErrorLines} from './utils/error-processing.js'
import {initBS} from './utils/browser-sync.js'
import {initWatcher} from './utils/source-watch.js'
import {executeTSCheck} from './utils/ts-check.js'

let buildErrors = [] // pay attention: it should be a const reference

console.log('[ Init build context... ]')

// prepare build context
let buildContext
try {
    buildContext = await esbuild.context({...buildConfig})
} catch (error) {
    console.error(error)
    process.exit(1)
}

// create and start dev server
bs.create('DevServer')
await initBS(bs, buildErrors)

// run source code watcher
initWatcher(doBuild)

let inBuild = false
let debounceTimer

function doBuild() {
    if (inBuild) {
        return
    }

    clearTimeout(debounceTimer)

    debounceTimer = setTimeout(async () => {
        inBuild = true
        buildErrors.length = 0
        console.log('[ Build... ]')

        bs.notify('Rebuilding...')

        let result = await executeTSCheck(buildErrors)

        if (result) {
            result = await executeBuild()
        }

        if (result) {
            bs.notify('🔄 Updating...')
        }

        bs.reload() // to render error banner

        console.log('')
        console.log(result ? '✅  Build successful.' : '⛔  Build failed.')
        inBuild = false
    }, DEBOUNCE_BUILD_TIME)
}

async function executeBuild() {
    console.time('[Re]build time')
    let result
    try {
        result = await buildContext.rebuild()
        if (result && result.errors.length > 0) {
            result.errors.forEach(error => buildErrors.push(error.text))
        }
    } catch (error) {
        fillErrorLines(buildErrors, error?.errors)
    }
    console.timeEnd('[Re]build time')
    return result && result.errors.length === 0
}

Decomposition

First, I prepare a build context and buildErrors array to use for each rebuild:

let buildErrors = [] // pay attention: it should be a const reference

let buildContext
try {
    buildContext = await esbuild.context({...buildConfig})
} catch (error) {
    console.error(error)
    process.exit(1)
}

Second, instantiate browser-check

// create and start dev server
bs.create('DevServer')
await initBS(bs, buildErrors)

Third, start observing the files in the ./src folder:

initWatcher(doBuild)

Then, back to up, watcher will start observing and on ready will call build. Build, in turn, will run Typecheck, and only if it passes will it run build itself. At the end, it will call browser-sync to report the completion of the process.

let inBuild = false
let debounceTimer

function doBuild() {
    if (inBuild) {
        return
    }

    clearTimeout(debounceTimer)

    debounceTimer = setTimeout(async () => {
        inBuild = true
        buildErrors.length = 0
        console.log('[ Build... ]')

        bs.notify('Rebuilding...')

        let result = await executeTSCheck(buildErrors)

        if (result) {
            result = await executeBuild()
        }

        if (result) {
            bs.notify('🔄 Updating...')
        }

        bs.reload() // to render error banner

        console.log('')
        console.log(result ? '✅  Build successful.' : '⛔  Build failed.')
        inBuild = false
    }, DEBOUNCE_BUILD_TIME)
}

DEBOUNCE_BUILD_TIME is defined as 500ms, so I expect runs to happen no more often in new builds — otherwise I need to increase the value or make de-bouncing more sophisticated. Like with AbortSignal or queue. But it’s already good enough.

Notifications and errors

Well, bun start, open localhost:3000 —

Project works

And, thanks to browser-sync, it displays a message and updates the page:

Top right corner

Wow, how professional!..

But I want it to displays errors:

Error in console

like something like this in a browser: an overlay with console messages passed through –

Error in browser

It is possible using https://browsersync.io/docs/options#option-snippetOptions — snippet which is able to inject strings in any HTML served.

During the build, I collect errors using the ESBuilds error structure or by intercepting the console for TS:

export async function executeTSCheck(buildErrors) {
    const check = await spawnTSC()
    let result = true

    if (check.error) {
        // ...
        errors.forEach((line) => {
            buildErrors.push(line)     // <-- add to global errors array
            console.log()              
            console.error(line)        // <-- display in console
        })
        result = false
    }
    return result
}

async function executeBuild() {
    let result
    try {
        result = await buildContext.rebuild()
        // ...
    } catch (error) {
        fillErrorLines(buildErrors, error?.errors)  // <-- add to errors
    }
    // ...
    return result && result.errors.length === 0
}

And in browser-sync.js later:

bs.init({
        server: OUTPUT_DIR,
...
        snippet: true,
        snippetOptions: {
            rule: {
                match: /<\/body>/i,
                fn: function (snippet, match) {
                    snippet += errorBanner(buildErrors) // <-- magic
                    return snippet + match
                }
            }
        },
    },

The default, prebuilt part of snippet is this thing:

Default browser-sync injection

It adds the reloading and syncing magic to the page. So I added error banner, if any:

// error-processing.js

const messagePrefix = '__build_error_message_'

export function errorBanner(buildErrors) {
    if (!buildErrors || buildErrors.length === 0) {
        return ''
    }

    // to safe html
    const errors = buildErrors.map(buildError => '<p>' +
        buildError.replaceAll('<', '&lt;')
            .replaceAll('<', '&gt;')
            .replaceAll('\n', '<br/>')
        + '</p>')

    return `
<style>
    #${messagePrefix},
    #${messagePrefix}::before,
    #${messagePrefix}::after,
    #${messagePrefix} *
    #${messagePrefix} *::before,
    #${messagePrefix} *::after {
      all: revert;                                 // <-- isolate styles
    }
    #${messagePrefix} {
       position: fixed;
       display: block;
       ...
    }
    ...
</style>
<div id="${messagePrefix}">
    <h1>Build error (at least ${buildErrors.length})</h1>
    ${errors.join('\n')}
</div>
`
}

The rest of the file is parsing the errors.

Since I can’t control all the formatters, I intercept the console, clear it of ANSI colours and convert it to plain text, more or less aligned. Dirty but effective:

Plus an error snippet

404

Everything seem all right? Well, almost.

Let’s remove ./dist folder and start bun start (serve) with error in source code. Oops, all my shiny error notifications are broken:

Ok in conole…

But wrong in a browser.

What’s going on? Where’s my beautifully stylised message?

Well, I’am running a type check before the build, because there’s no point in building if the types are broken, and if the build fails, no final HTML is created at all. And browser-sync reports with Cannot Get %HTML% virtual page without proper injecting of styles and scripts. CSP headers are broken. Crap.

So I need to add a middleware —

// browser-sync.js  
const content_404 = '<!DOCTYPE html><html lang="en"><body>OOPS, file not found!</body></html>'
...
bs.init({
      server: OUTPUT_DIR,
      ...
      snippetOptions: {
        ...
      },
      callbacks: {  // <-- this one
          ready: function (err, bs) {
              bs.addMiddleware('*', (req, res) => {
                  res.writeHead(404, 
                    { 'Content-Type': 'text/html; charset=UTF-8' })
                  res.write(content_404)
                  res.end()
              })
          },
      }
  },

and let the browser-sync do its usual injecting magic. And now it works. Finally.

Bonus: WebComponents

Well, our target applications are non-framework, custom JS-intensive (or JS-less, it depends on the situation). To have a rich UI/UX — if we need it at all –we can use CSS frameworks like Tailwind for styles and WebComponents for incapsulated UI elements.

I’ve added it to the Dashboard page for an example.

<!-- dashboard.html -->
<link href="css/gauges.scss" rel="stylesheet"/>
...
    <div id="gauges">
        <div class="loading">Loading web components...</div>
        <div class="gauge">
            <progress-ring duration="1000" percentage="30"></progress-ring>
            <div class="buttons">
                <button id="buttonOne">30%</button>
                <button id="buttonTwo">60%</button>
                <button id="buttonThree">90%</button>
            </div>
        </div>
        <div class="gauge">
            <progress-ring duration="2000" percentage="60" round-linecap="true"></progress-ring>
        </div>
        <div class="gauge">
            <progress-ring disable-digits="true" duration="500" percentage="90">
                <p class="completed-count">9/10<br/>Complete</p>
            </progress-ring>
        </div>
    </div>
...
<script defer="" src="https://unpkg.com/progress-ring-component@1.0.36/dist/progressring/progressring.esm.js" type="module"></script>

then

// gauges.scss
#gauges {
  border-radius: 18px;
  margin: 0 auto;
  display: flex;
  flex-flow: row nowrap;
  align-items: flex-start;
  justify-content: center;
  min-width: 650px;
  width: fit-content;
  position: relative;
  transition: all 1.5s ease-in-out;

  ...
}

and some logic for detecting the readiness of WebComponents and assigning handlers:

export function initGauges() {
    const gaugesContainer = document.querySelector('#gauges')

    if (!gaugesContainer) {
        console.error('No gauges found.')
        return
    }

    const componentsObserver = new ResizeObserver((c) => {
        // yes, i know, it is rather stupid way to detect if loaded
        // but it works =)
        if (c[0]?.contentRect?.height > 200) {
            console.log('Web components loaded')
            assignButtons()
            gaugesContainer.classList.add('loaded')
        }
    })

    console.log('Size watcher attached', gaugesContainer)
    componentsObserver.observe(gaugesContainer)
}

function assignButtons() {
    const ring = document.querySelector('progress-ring')
    const buttonOne = document.querySelector('#buttonOne')
    const buttonTwo = document.querySelector('#buttonTwo')
    const buttonThree = document.querySelector('#buttonThree')

    buttonOne?.addEventListener('click', () => {
        ring?.setAttribute('percentage', '30')
    })

    buttonTwo?.addEventListener('click', () => {
        ring?.setAttribute('percentage', '60')
    })

    buttonThree?.addEventListener('click', () => {
        ring?.setAttribute('percentage', '90')
    })
}

Voila –

Dashboard with webcomponents

GitHub repository: https://github.com/lexey111/insane-nano-builder

Live page: https://lexey111.github.io/insane-nano-builder/

What’s next? Well with this rather modular structure I can replace, e.g., ESBuild with https://rolldown.rs/ and experiment with it.

Bonus: https://github.com/lexey111/insane-nano-preact and live page: extended with Preact and some post-build analyses:

Build stats

It’s not always convenient to work with raw JS/TS when you need to create a more or less complex UI wrapper. That’s why I added Preact support for convenient work with components and compositions.

Thanks for reading. I hope it helps you create small, well-targeted builds for the frontend part of tiny projects.