Here’s the code I discuss in this post, the README provides a more technical discussion.
New controllers for the web are a reality. Game controllers, their variations and ability to facilitate immersion, have been part of the main stream culture for a few decades. The variety of interfaces we have for musical instruments is probably so natural that it could be considered a defining aspect of what makes any type of instrument unique. For interacting with web-browser applications, there is little diversity: a mouse, a keyboard, a touch screen. Maybe I could include a handful of accessibility tools, but that is specific to a niche group of users. With a bunch of new APIs: MIDI, HID, USB, Serial, Bluetooth, Sensor, and Gamepad, there’s now the possibility to build web applications that expose a control interface that can be altered by events coming from these serial communication systems.
In order to gain an understanding of the complexity behind building a device and communicating with the browser, I created a small project and detailed what I discovered along the way. In short, it’s quite easy to get something working, so anyone can have a novel interface and immediately use it with a web app. The project depends on a Raspberry Pico as the serial peripheral, but I believe any microcontroller that can write to serial should be able to do the same. The goal was to use 2 potentiometers to control 2 different web audio oscillator nodes, then analyse the result both visually and auditorily.
Working with physical devices and serial data can be complex. They present difficult problems that, without specialised hardware or knowledge, might be solved only by trial and error. I’m not sure if it’s a system-related issue, but when switching the MCU between the IDE and the browser, I often need to disconnect the MCU completely, then reconnect before I could access it. Additionally, there were times where the connection would be lost without any explanation. There were a few extra ideas I hadn’t explored before, like parsing messages in byte-forn with things like starting bytes and bitwise operators to calculate the values. Noise is also a reality, the values being emitted aren’t stable due to the nature of electronic circuits and ADCs, so I implemented a noise threshold that needs to be exceeded before a newly received value is considered intentional.
There are some awesome oscilloscope libraries available, so I selected one at random, and although it needed some rework to be usable in my project, this was fairly trivial and the result enabled the controlled waves to be visualised.
It was immediately clear to me there’s a lot of potential with this API enhanced by the affordable technology now on the market. I created a subreddit to explore the Web Serial API further, hopefully it can help generate interest. I imagine all kinds of new ways to interact with educational web apps, creative collaboration tools, and immersive experiences. I’m planning to build some boilerplate tools to help with the rapid prototyping and ideation process, and possibly develop some ideas for project challenges and friendly competitions using new controller designs. If you’re interested, join in the discussion in the subreddit, or reach out to me directly.