Usage</strong></th>	Client/Server</strong></th></tr></thead>
Settings for a single device</td>	Client, only needed on that device</td></tr>
Social Media post</td>	Server, needs to be stored centrally so many users can access it</td></tr>
QR code scan</td>	Client, most devices are powerful enough to run QR code scanning code without needing cloud support</td></tr>
Access control</td>	Server, the client side is not a trusted environment and therefore cannot perform access control in a trustworthy way</td></tr>
Running an LLM</td>	...</td></tr> </tbody></table> When a network connection is faster than doing the computation on the client</a></h3> Consider the last example in the table. With our rule, we would run it on the client, as technically it can</em> be run on the client, but on most devices it would be far faster to run it on a server and wait for the dreaded network connection. If, for a significant proportion of your users, it is faster to do a computation on the server than on the client, then its best to do it on the server if possible. Another approach if this proportion is not your entire user population, is to start the operation on the client and server in parallel, and when one finishes abort the other.</p> How to make the network more bearable</a></h2> Sometimes you have to fetch across the network, like accessing a piece of shared data or something that needs access control. When this happens, we can still make some optimisations to make it feel more user-friendly. Here are some things you can do to improve your user experience for fetching server-side data:</p> Optimistic updates After making a change to server side state, update it on the client side assuming it succeeded and if it fails then revert it.</li> </ul> </li> Persistent caching Cache server-side data between page loads, so data is shown immediately (state data is better than no data)</li> </ul> </li> Periodic sync When offline, queue up changes to be made to the server-side and sync this in the background when back online</li> </ul> </li> Validation Validate data to avoid undefined behaviour when invalid data is given, either because of an actual bug or a proxy (like on school or corporate networks)</li> </ul> </li> Graceful error handling Keep showing stale data even when an error occurs so the user can work with the stale data until the issues stop</li> </ul> </li> Exponential backoff on retries If an error is temporary it might resolve itself after some period of time, so keep trying a certain number of times. Use exponential backoff to avoid putting unnecessary load on the server.</li> </ul> </li> </ul> This is not easy to implement yourself. You may want to use a data-fetching library, sometimes called an async state manager. This library will be responsible for syncing server-side state with a client-side "store", which your app can deal with, so you don't need to worry about the network as much. I personally use TanStack Query</a> for web apps, and I've heard good things about SWR</a>. I don't currently have any recommendations for native. Also for things like periodic sync and persistence you may need to do some extra configuration. For validation of network requests on the Web, I use zod</a>.</p> How to use OAuth PKCE in client-side JS 2023-05-22T00:00:00+00:00 You need to use some API that requires authentication. It doesn't have an SDK. 2 years ago, I was in this position, I started building Timetabl</a>. The API I was using uses OAuth</a>, a protocol for authorising third-party applications, to access a user's data. I started digging through articles about OAuth. Looking at the available libraries for client side OAuth, I didn't seem to have many options. Most JS client libraries were either not actively maintained, didn't support PKCE (Proof Key for Code Exchange), didn't run in the browser, or were too complicated for me to understand.</p> If you don't understand much about OAuth, or just want a refresher, check out this article</a> for an explainer.</p> What not to do</a></h2> Stubborn, I dug into the RFCs for several weeks, and tried to wrap my head around the complexities of OAuth. In around a month I got a basic prototype implementation working. As my application grew, I started to realise that ~90% of my bugs came from that OAuth implementation. As the complexity increased it became almost impossible to maintain. Eventually, I decided to start looking for OAuth libraries again.</p> A library that makes OAuth easy</a></h2> After some research, I found badgateway/oauth2-client</a>. It is an excellent library that is actively maintained. I'm surprised it dosen't have a bigger community, so give it a ⭐ and contribute if you can. This library has unit tests, and is fairly well-documented.</p> Registering your application</a></h3> Go to the dashboard of your OAuth provider, and create a new application. You will see a page like this:</p> You can ignore the "Client Secret" field, as we will be using PKCE</a> to authenticate. Note down the "Client ID" and "Redirect URI" fields as we will need them for later.</p> Setting up the library</a></h3> Install the library with npm:</p> `npm</span> i @badgateway/oauth2-client </span></code></pre> Next, setup the client class:</p>`import </span>{ </span>OAuth2Client</span>, </span>generateCodeVerifier </span>} </span>from </span>"</span>@badgateway/oauth2-client</span>"</span>; </span> </span>// Replace the following values with your own </span>const </span>client </span>= new </span>OAuth2Client</span>({ </span> server: </span>"</span>https://my-auth-server/</span>"</span>, </span> clientId: </span>"</span>...</span>"</span>, </span> tokenEndpoint: </span>"</span>/api/token</span>"</span>, </span> authorizationEndpoint: </span>"</span>/api/authorize</span>"</span>, </span>}); </span></code></pre> The values for server</code>, tokenEndpoint</code>, and authorizationEndpoint</code> will be in your OAuth provider's documentation. The clientId</code> is the "Client ID" you noted down earlier.</p> Logging in</a></h3> Add a login button to your page, and add the following code to the click handler:</p> const </span>codeVerifier </span>= </span>await </span>generateCodeVerifier</span>(); </span>// You should really dynamically generate a random string, but this is fine for a demo </span>const </span>state </span>= </span>"</span>some-string</span>"</span>; </span> </span>document.location = </span>await </span>client</span>.</span>authorizationCode</span>.</span>getAuthorizeUri</span>({ </span> </span>// The redirect uri you noted down earlier </span> redirectUri: </span>"</span>https://my-app.example/</span>"</span>, </span> </span>state</span>, </span> </span>codeVerifier</span>, </span> </span>// see your OAuth provider's documentation for available scopes </span> scope: [</span>"</span>scope1</span>"</span>, </span>"</span>scope2</span>"</span>], </span>}); </span> </span>// Save the code verifier and state to the localStorage </span>localStorage</span>.setItem(</span>"</span>codeVerifier</span>"</span>, </span>codeVerifier</span>); </span>localStorage</span>.setItem(</span>"</span>state</span>"</span>, </span>state</span>); </span></code></pre> This code redirects the user to the OAuth provider's login page. After the user logs in, they will be redirected back to your application. This code will extract the code</code> and state</code> from the URL, and exchange the code</code> for an access token.</p> // Get query string </span>const </span>query </span>= </span>Object</span>.</span>fromEntries</span>( </span> new </span>URLSearchParams</span>(window.location.</span>search</span>).entries() </span>); </span> </span>// Error check </span>if </span>(</span>query</span>.</span>error</span>) { </span> </span>// Handle error </span> </span>// ... </span> </span> </span>// Logout </span> </span>localStorage</span>.setItem(</span>"</span>loggedIn</span>"</span>, false); </span>} </span> </span>// If the server returned an authorization code, attempt to exchange it for an access token </span>if </span>(</span>query</span>.code) { </span> </span>const </span>codeVerifier </span>= </span>localStorage</span>.getItem(</span>"</span>codeVerifier</span>"</span>); </span> </span>const </span>state </span>= </span>localStorage</span>.getItem(</span>"</span>state</span>"</span>); </span> </span> </span>const </span>oauth2Token </span>= </span>await </span>client</span>.</span>authorizationCode</span>.</span>getTokenFromCodeRedirect</span>( </span> document.location </span>as </span>any, </span> { </span> </span>/** </span> * The redirect URI is not actually used for any redirects, but MUST be the </span> * same as what you passed earlier to "authorizationCode" </span> */ </span> redirectUri: </span>config</span>.</span>redirect_uri</span>, </span> </span> </span>state</span>, </span> </span>codeVerifier</span>, </span> } </span> ); </span> </span> </span>// Save the token object to localStorage </span> </span>localStorage</span>.setItem(</span>"</span>token</span>"</span>, JSON.stringify(</span>oauth2Token</span>)); </span>} </span> </span>// Clear query string </span>window.history.</span>replaceState</span>({}, null, </span>location</span>.pathname); </span></code></pre> Logging out</a></h3> To log out, simply remove the token from localStorage and set loggedIn</code> to false</code>:</p> localStorage</span>.removeItem(</span>"</span>token</span>"</span>); </span>localStorage</span>.setItem(</span>"</span>loggedIn</span>"</span>, false); </span></code></pre> Making requests</a></h3> To do this we need to setup the FetchWrapper</code>:</p> const </span>fetchWrapper </span>= new </span>OAuth2Fetch</span>({ </span> </span>client</span>, </span> </span> </span>getNewToken</span>: () </span>=> </span>{ </span> </span>// This method will only be called when refreshing the tokens fails </span> </span>// You should log the user in again here </span> </span> </span>// `return null` if you want to fail this step </span> }, </span> </span> </span>storeToken</span>: (</span>token</span>) </span>=> </span>{ </span> </span>// This method will be called when the token is refreshed </span> </span>// You should save the token to localStorage here </span> </span>localStorage</span>.setItem(</span>"</span>token</span>"</span>, JSON.stringify(</span>token</span>)); </span> }, </span> }, </span> </span> </span>getStoredToken</span>: () </span>=> </span>{ </span> </span>// This method will be called to get the token from storage </span> </span>return </span>JSON.parse(</span>localStorage</span>.getItem(</span>"</span>token</span>"</span>)); </span> }, </span>}); </span></code></pre> To send a request, simply call fetchWrapper.fetch</code> which behaves similarly to plain old fetch</code>:</p> const </span>response </span>= </span>await </span>fetchWrapper</span>.</span>fetch</span>(</span>"</span>https://my-api.example/</span>"</span>); </span> </span>// Obviously do some error handling too... </span> </span>const </span>data </span>= </span>response</span>.</span>json</span>(); </span></code></pre> This FetchWrapper</code> will automatically refresh the token when it expires, and retry the request. It will also handle sending the access tokens.</p> Is it secure to store the token in localStorage?</a></h2> The main issue with localStorage is that it can be accessed by JS. In modern applications, you will often have thousands of dependencies, and if any of them are compromised then you could end up with an XSS attack. You can partially mitigate this risk with a strong Content Security Policy. The JS supply chain is quite resilient to compromised packages, with them being taken down quickly, and the community being notified. It is also very hard to pull off such a targeted attack, through a compromised package.</p> However, let's assume that your app is already compromised through XSS. Even if the token was not stored in localStorage, there would be some other mechanism to make API requests (possibly through a proxy server which stores tokens as HTTP-only cookies), and the attacker could still use that by sending requests from the context of your app.</p> In our case, the attacker will be able to access the token from localStorage</code>, and possibly use it even when the user does not have the app open. However, there is a measure against this called refresh token rotation</strong>.</p> Refresh Token Rotation</a></h3> Refresh token rotation ensures that only one party is using the tokens at one time. This prevents attackers from stealing the token and using it outside the application. It goes hand in hand with short-lived access tokens.</p> Short-lived access tokens</a></h4> Access tokens should be short-lived, and expire after a short period of time. This could be anywhere from 5 minutes to 1 hour. This means that even if the attacker steals the token, they can only use it for a short period of time, before they have to use the refresh token to refresh it.</p> The security advantage of refresh token rotation</a></h4> When the client requests a new access token, it also receives a new refresh token. If a malcious client uses the same refresh token to retrieve a new access token, the server will detect that the refresh token has been used twice, and will revoke all refresh and access tokens. This means that the attacker cannot use the refresh token outside the application, as it will be revoked as soon as the user opens the app.</p> Read more about refresh token rotation here</a>.</p> My approach to web app development 2023-04-11T00:00:00+00:00 This blog post outlines the way I build web applications. As the title suggests, this is my</em></strong> approach, and is what works for me. This is a heavily opinionated blog post, but I hope that at least some concepts suggested here will be helpful for others.</p> Web apps vs. websites</a></h2> The first thing to clarify is that web apps</strong> should be built differently from websites</strong>. Web apps</strong> are focused on functionality, while web sites</strong> are focused on content. Web apps</strong> may have a small amount of content, and web sites</strong> may have a small amount of functionality, but the main focus is different. Additionally, web apps</strong> are designed to be used more than once, while for web sites</strong> they may or may not be used regularly. This distinction is important to consider in order to optimise the user experience.</p> In reality, there really is a spectrum between these two extremes. We will be focusing on the web app</strong> extreme of this spectrum.</p> </p> To server-render or not to server-render?</a></h2> A lot of the development happening with frameworks right now is about server rendering. By server rendering I'm referring to any type of rendering that happens remotely (not on the client). This definition includes SSR</strong> (server-side rendering), SSG</strong> (static site generation), and ISR</strong> (incremental static regeneration). On the other hand, CSR</strong> (client-side rendering) is rendering that happens on the client, which is the 'default' for most frameworks (React, Vue, Angular).</p> There are three combinations of server-rendering and client-rendering (excluding when there are none):</p> </p> Server-rendered (no client-rendering)</a></h3> This approach is reminiscent of frameworks such as PHP/Django. The server renders the HTML and sends it to the client. The client then receives the HTML and displays it. The client does not render anything. Application is entirely stored on the server, through cookies or sessions. Data will not update without a refresh. Mutations are performed through forms which submit POST requests to the server, and cause the page to refresh.</p> Progressive Enhancement</a></h3> This is the approach taken by Next.js, Remix, etc. The server renders the HTML and sends it to the client. The client then receives the HTML and displays it. This is good for performance and SEO, as data is displayed without needing JS to load. The client then performs a process called 'hydration', which means that it takes the HTML and turns it into a fully interactive application. The application state is stored on the client. Data will update without a refresh (with proper data fetching libraries). Mutations can be performed without refreshing the page.</p> Client-rendered (no server-rendering)</a></h3> The server sends the client a blank HTML page, which references some scripts which render the page. The application state is stored on the client. Data will update without a refresh (with proper data fetching libraries). Mutations can be performed without refreshing the page. Often, the app uses client side routing, which means that navigation between pages happen without a refresh. These apps are often large, as the entire app is downloaded to the client, rather than a single page.</p> After looking at these three, Progressive Enhancement seems like the most obvious choice. In progressive enhancement, data is fetched on the server, and then hydrated on the client. This means that the user gets the best of both worlds: the performance and SEO benefits of server-rendering, and the interactivity of client-rendering.</p> PWAs</a></h2> Progressive Web Apps (PWAs) are applications built using web technologies to deliver experiences that rival those of native apps. A key capability of PWAs is that they use service workers. Service workers are a type of web worker that runs in a separate thread, and among their capabilities is the ability to cache and intercept network requests.</p> Web applications should behave like native apps. Native apps don't download their entire codebase from a server every single time</em> they are opened. Native apps don't need to be connected to the internet to work. If web apps are to be viable alternatives to native apps, they need to have these capabilities.</p> The web applications I build rely extensively on service workers. On first load, the web app is downloaded from the server and cached. On subsequent loads, the web app is loaded exclusively from the cache. When there is an update, the new code is downloaded in the background. When the user refreshes the page, the new code is loaded from the cache.</p> For this to work, there needs to be some kind of client-side rendering, as the app is largely decoupled from the server. This automatically rules out the PHP-style approach, with no rendering on the client.</p> Progressive Enhancement or CSR?</a></h2> The question now is whether to use progressive enhancement or client-side rendering. For the reasons outlined above, progessive enhancement will often provide the best user experience. However, for PWAs, progressive enhancement has diminishing returns. Consider that for devices supporting PWAs, the client only loads the app once, and then it is cached. This means that the benefits of progressive enhancement are only gained on first load. And since apps are used numerous times, these benefits are not as important, as for the vast majority of times, progressive enhancement has no benefit. caniuse.com</a> estimates that 96.4%</strong> of users globally support service workers. As nice as it would be to support the remaining 3.6%, for most developers, the cost of supporting these users is not worth it.</p> Architectural Divergence</a></h3> As outlined in Remix's</a> article, architectural divergence leads to extra complexity, which leads to poor DX (developer experience). The architectural divergence Remix is talking about is about the divergence between SSG and CSR. Remix's solution is to fetch data on the server through SSR only</strong>. This is a great solution for websites, but as outlined above it doesn't work for PWAs. The architectural divergence in progressive enhancement is the initial fetching on the server, and the partial updates on the client. This leads to needing to write logic and data fetching code that runs both on the server and client, leading to an increase in complexity caused by architectural divergence.</p> As a result, I opt to go for CSR-only for PWAs. This removes the additional complexity of data fetching on the server, which leads to a better DX. Progressive enhancement may be an ideal solution, but unless you have the time and resources to support the 3.6% of users who don't support service workers, it is not worth it.</p> Making SPAs work well</a></h2> There are a lot of issues with SPAs. Here are the two most common issues:</p> SEO</strong> - Search engines can't crawl SPAs.</li> Performance</strong> - SPAs are slow to load, and slow to run.</li> </ul> SEO</a></h3> Many people point to SEO as an issue for SPAs. However, we are talking about using SPAs for apps, not websites. Since apps are about functionality, not content, there is not much for the search engine to crawl in your app anyway. As for discoverability, you should build a website landing page to promote your app, using SSR or SSG. This landing page should be discoverable by search engines, and should link to your app.</p> Performance</a></h3> This isssue has two sub-issues: initial load time, and runtime performance.</p> Initial load time</a></h4> SPAs tend to be quite large, as they contain the entire app and all of its pages. This can easily lead to bundle sizes in excess of several megabytes. However, since we are building PWAs, this download only needs to happen once, as it is then cached. Additionally, through code-splitting, we can ensure that on first launch, only the code needed to render the current page is downloaded. This makes initial load time much better. And even if after this, load times are still bad, it only needs to happen once, so the impacts are minimal. After this initial load, service workers should start caching the rest of the app in the background.</p> Runtime performance</a></h4> Since the app is using CSR, all of the rendering logic happens on the client. This moves the bottleneck from the network to the CPU. This is usually good, since for most devices it is faster to do things locally than to wait for the network. However, for extremely slow devices, this can be an issue. There is no easy answer to this problem, but generally performance issues are caused by poorly written code, and can be solved by refactoring. Ultimately, you should tailor your apps to work well for most of your users. You should collect performance metrics such as Core Web Vitals, and use them to make informed decisions about what to optimise. If most of your users are on devices that cannot support an SPA approach, than you should consider another approach.</p> Data fetching</a></h2> One of the key principles I use when building apps is to have some data available as soon as possible, and try to keep that data up to date. Having data available as soon as possible, can be accomplished by having a cache, so that the data can loaded from the cache quickly. Keeping this data up to date could be accomplished by refreshing the data on a certain interval, and doing it whenever the user refocuses on a page. This can be quite hard to implement, so libraries such as SWR</a> and TanStack Query</a> exist to make this easier.</p> Conclusion</a></h2> In this article, I outlined the pros and cons of the three main approaches to building web apps: SSR, CSR, and progressive enhancement. I then outlined the pros and cons of using progressive enhancement for PWAs. I concluded that for PWAs, CSR is the best approach, as it removes the complexity of data fetching on the server, and leads to a better DX. I also outlined some of the issues with SPAs, and how to solve them. Finally, I outlined how to keep data up to date. Keep in mind that this is just how I build apps, and you should do what leads to the best user experience for your users.</p>

How we placed first in RoboCup Junior NSW State

2026-11-08T00:00:00+00:00

Over the past 2 years, I've led a team building robots that play soccer autonomously to compete in the RoboCup Junior competition. In August, our team placed first in the Opens Soccer division at the RoboCup Junior Australia NSW State competition. We did not pursue nationals due to being busy with our HSC exams. My role primarily involved working on the software and firmware for the robot, as well as reviewing and integrating hardware and software components to create a functioning system. In this blog post I'll discuss my contribution to this project as well as some technical details that could help future teams.

But first, here is a highlights video from the state competition:

</iframe> </div>

Overall, our robots scored 20 goals throughout the competition while only conceding 10. Some things you may notice from this video:

The robots can both rotate and translate, tending to "drift" laterally to receive the ball</li>
Our robots are not particularly fast and in some cases significantly slower than the other robots, so speed was not our main advantage</li>
Many times either our robots or the opposing teams' robots are not fully functioning</li> </ul>
I will explain both the factors leading towards our success and improvements that we can make and in many cases had already made partial progress on later in this post. Before that, I will explain the overall robot design to contextualise these points.
Robot architecture</a></h1>
This poster describes our overall robot design. My primary role was working on the control software, computer vision pipeline and PCB (printed circuit board) design but some other aspects of the robot are described here for context. Notably:
- When the robot possesses the ball it is out of view of the camera, so a VL53L0X time of flight (ToF) sensor allows the robot to ensure it really does have possession of the ball. When the ball enters the dribbler zone, it shortens the distance detected by the ToF sensor allowing the robot to register that it has successfully captured the ball. The graph on the right in the above image shows the measurable decrease in distance measured by the sensor when the ball enters the dribbler. </li> </ul>
 PCB design</a></h1>
 </div>
 I designed the motor controller PCBs for these robots in KiCAD. These motor controller PCBs perform closed-loop motor control for the mecanum wheels, control the dribbler and distribute power from the battery with a fuse in case of short circuit.
 Our robots use four individually controlled motors for the mecanum wheels which are connected to motor drivers (which each need 2 connections from the motor terminals to the driver, and PWM and direction connections from the driver to the MCU) and each of these motors have 4 further connections from the quadrature encoders reporting motor rotation to the MCU. That gives a total of 32 connections for locomotion alone.
 Hence, the reasons I designed the PCBs were:
 
 To avoid needing to manually wire these connections during assembly, affording us the ability to rapidly assemble and disassemble portions of the robot to debug issues</li>
 For consistency in pinout between the two robots</li>
 For greater reliability: jumper wires can come loose or snap at joints if soldered</li> </ul>
 The first revision of this board as the first PCB I had ever designed, and the boards on our state competition robots were the second. To reduce the opportunity for error, this PCB uses breakout boards such as a Raspberry Pi Pico (a development board for the RP2040 MCU) and DRI0044 (a breakout for the TB6612FNG motor driver IC) boards which each hold those difficult-to-solder surface-mount ICs as well as their support circuitry.
 These reliability improvements are a major theme of our team's work in 2026, and our success in the state competition was largely due to this: our robots worked more consistently, and when they were working they could quickly gain possession of the ball to easily score a goal.
 Firmware</a></h1>
 The Raspberry Pi Pico on the aforementioned PCB runs Rust firmware I wrote. This firmware takes movement commands from the main Raspberry Pi board over USB, determines motor speeds needed to achieve that movement, and performs closed loop control to keep motors at that speed. It uses the Embassy</a> framework which provides a hardware abstraction layer and async runtime.
 The diagram above describes the general architecture for the firmware. The kinematics and closed-loop control portions of this system are rather standard in mechatronic systems, however this level of closed-loop control is rare in RoboCup Junior. Recall that the robot has 3 axes of motion: longitudinal, lateral and yaw. Inverse kinematics takes a velocity for each of these 3 axes and converts it into appropriate mecanum wheel rotational velocities to achieve that chassis velocity. Forwards kinematics conversely takes the reported rotations from the motor encoder and converts it into a chassis velocity. The combination of these two allow us to make any planar movement on the robot.
 The parts of the system I have described so far were already in place in the original MicroPython implementation I wrote in 2025. The reason I rewrote this firmware was related to the interface between the Raspberry Pi Pico (the microcontroller on our motor controller PCB) and the Raspberry Pi 4 (the main "brains" of the robot, processing camera data and making decisions). These two components are linked by a micro-USB cable, carrying power and data. Vanilla MicroPython does support taking input over USB (by sys.stdin</code> or input</code>) but this same serial channel is used for the MicroPython REPL and there is no native support for separating out serial channels or having any serious control over USB configuration. Overall this led to a more brittle system that was difficult to debug. I also made the mistake of trying to have odometry be reported asynchronously from velocity commands, but the MicroPython setup is really not designed for this in mind so I had lots of reliability issues.
 Instead, in the Rust rewrite I created two separate USB communication device class (CDC) channels: one for debug logging and one synchronous request/response interface for sending commands. The Embassy embedded framework made this straightforward as I had complete control over USB packet processing on the device. In the diagram above you see COBS decode/encode blocks. COBS is a packet framing scheme (basically an encoding that allows both sides to know where commands start and end) so you may be wondering why a packet framing is needed when USB already involves sending packets. For simplicity, I chose to use USB CDC ACM (basically a serial interface) for the command channel, and this device class abstracts over the packets and rather represents data sent as a stream of bytes. Hence, a framing scheme is needed to delineate this stream.
 The advantage of using CDC ACM is that on the Raspberry Pi side, the device appears as a pair of standard TTYs which can be read and written to with pyserial</code>. In retrospect, I consider using CDC ACM to be a mistake. Alongside being conceptually inelegant, this scheme means data sent and received passes through the Linux kernel's TTY layer (which is highly non-deterministic) which has the right to buffer data arbitrarily. Despite my attempts to write resilient retry logic to resend commands if no response is seen, competition day exposed new link failure modes that our extensive testing could not find. If I were to build this interface again I would create a custom device class and write a user-space driver for it, to have greater visibility into hotplug and disconnect events.
 Control software</a></h1> Multi-process architecture</a></h2> I wrote the control software that runs on Raspberry Pi OS. The architectural diagram below is from the repository README: As shown, there are four different processes running in parallel: camera_streamer</code>: This process accesses camera frames using the picamera2</code> library, writing them to a numpy</code> array mapped in shared memory. I wrote it as a separate process because picamera2</code> is distributed by the Raspberry Pi OS package repositories and is dependent on shared libraries distributed that way. As I wanted to use virtual environments to manage dependencies for the controller</code> process this had to be separated. </li> controller</code>: This is the main control process, sending commands to the motor controller board and reading in camera frames from camera_streamer</code>. There is no synchronisation between camera_streamer</code> and controller</code> which means that there can be tearing artefacts if this process reads while the streamer is writing but this isn't an issue for our computer vision workloads. The robot strategy is organised in a behaviour tree using the pytrees</code> library. </li> serial_debug</code>: This is a process spawned by the controller</code> process to log data received through the USB debug channel to Rerun. This is primarily useful for troubleshooting motor encoders or tuning the PID loop. </li> ui</code>: This is kept separate from controller because of the need for the gpiozero</code> library, which also poses problems when installed from a virtualenv. It handles button presses from the physical user interface on the robot (used for starting, stopping, changing goals, and powering down the robot) an publishes events over a ZeroMQ channel for the controller</code> package to read. This process also controls two important status LEDs: a pulsing LED to show when the Raspberry Pi has completed boot, and another LED indicating which goal the robot is configured to score into. </li> </ul> camera_streamer</code>, controller</code>, and ui</code> are all launched by their respective systemd</code> units on boot. This also enables these processes to be restarted individually, and for their logs to be monitored using journalctl</code>. Behaviour tree</a></h2> The behaviour tree is the decision making engine for our robots, and determines at a high level what the robot should do: for example whether the robot should move towards the goal or towards the ball. I chose behaviour trees over other approaches such as finite state machines mainly because they can very naturally represent a notion of cascading priorities, where a higher priority task (like gaining possession of the ball) can override a lower priority one (like searching for the ball). In the above diagram, every node with black outline is a behaviour, which can briefly execute some code when they are "ticked". After being ticked, behaviours return RUNNING</code> if the task they represent is ongoing or SUCCESS</code> or FAILURE</code> upon completion. Behaviour trees have three main composites (these are nodes in the above diagram that point to multiple child nodes to combine their behaviours): Parallel (parallelogram in the above diagram): will tick every node under it every time it is ticked (overall effect of doing multiple tasks at once)</li> Sequence (The rectangle in the diagram): each time it is ticked will tick each node under it in succession until any of its child nodes return RUNNING</code> or FAILURE</code> (overall effect of doing one task after another)</li> Selector (The octagon in the diagram): each time it is ticked it will try each child node under it until any of its child nodes return RUNNING</code> or SUCCESS</code> (overall effect of trying different tasks until one of them works)</li> </ul> Despite the above behaviour tree being rather basic, it was reliable and this contributed to our robots' performance. In early stages of robotics development, high-level planning software is seldom a bottleneck, rather it is the hardware and low-level control software that determines a system's success. For almost all of our two year development, improving the reliability and performance of the lower levels of our stack was a better use of time than implementing a sophisticated strategy, because these strategies are contingent on reliable and effective hardware and control. Computer vision and localisation</a></h2> Our robots' camera setup is unconventional, as we use a wide-angle forward facing camera rather than a 360-degree mirror setup like most RoboCup Junior robots. The purpose of a RoboCup computer vision system is to identify the locations of the ball, goals and possibly field lines or even other robots. More sophisticated strategies become possible when robots can determine the physical relative coordinates of objects (in real-life units e.g. in millimetres), which is called localisation. Localisation is a prerequisite for proper trajectory planning and navigation, coordination between robots, and deduction of where the ball is (by Bayesian inference, for example). I wrote our robots' computer vision system, and localisation by fusing camera data and odometry was my ultimate goal. However, as mentioned earlier, these more sophisticated techniques are contingent on having basic hardware functionality which I was also responsible for. In the time I had, I only implemented a partial localisation system which I will describe below. Localisation systems are really only seen at the international level in RoboCup Junior. The main alternative approach I have heard of relies on using LiDAR to locate the field walls. My approach is purely camera-based and not even requiring an additional mirror, thus bringing the potential to make localisation systems more accessible for less experienced teams. This is an unprocessed camera frame from the wide angle camera. If you look at the white field lines, you will notice that they curve slightly due to a slight fisheye effect. On this raw frame, HSV colour thresholds are used to identify objects, since the RCJA soccer rules standardise the goals to be yellow and cyan, the ball to be orange and field lines to be white with particular dimensions. Some teams use more complex methods such as a machine learning model to do this, but I found simple colour thresholding to produce acceptable results at our stage of development. I also did further tuning after this screenshot is taken. The purple magenta lines drawn are detected by a sequence of white thresholding, canny edge detection, and then a probabilistic Hough line transform to detect field lines. At this step in the pipeline, you can see a lot of garbage lines detected above the field. These lines, alongside any other detections outside the field, will be filtered out in a later step of the process. This is a view of the camera after undistortion. In controller/calibrate.py</code>, a simple script calibrates the camera by fitting a model of fisheye distortion to the coordinates of a flat checkerboard pattern. The distortion coefficients, image resolution and camera matrix are written to a JSON file for later use. The undistorted image above was produced using these coefficients, and as you can see, the white lines are no longer curved as in the original image. Fully undistorting the image is very computationally expensive and is only done for debugging and visualisation purposes. Instead my code uses cv2.fisheye.undistortPoints</code> to undistort only specific coordinates needed to be undistorted, as identified in the previous step. This undistortion step is required to be able to accurately determine real world coordinates of objects in the image. This is probably the most interesting step and is one that I have not ever seen in RoboCup Junior before. In this step, I project points on the field from the camera frame into real-world coordinates in millimetres, relative to the camera. This is effectively partial localisation, and in our control software this enables a much more effective ball acquisition tactic. I call this partial localisation because it can find the relative coordinates of any object on the field relative to the robot, but full localisation would give positions of objects in a shared coordinate frame relative to the field itself. The above is a visualisation of all points being projected this way, but again the deployed code only projects the points that are needed for efficiency's sake. The principle that makes this projection possible is related to degrees of freedom. In the camera frame seen by a robot, a pixel has two degrees of freedom (row and column in image). For an object on the field, there are also two degrees of freedom (since the objects only move on the plane, they cannot be in the air). Thus, there is a mathematical function that transforms from one space to the other. More specifically each pixel corresponds to a point on an image plane (an imaginary plane distance 1 away from the camera). cv2.fisheye.undistortPoints</code> can give us coordinates on this plane. Draw a ray from the camera origin through this point on the image plane, extending it until it hits the field. The robot is a fixed height in millimetres above the field (the robot moves only in planar motion, it never travels up or down), so this intersection point between the field and this plane in the field can be easily found. Since the distance from the camera to the field is in millimetres, the coordinates of the intersection point on the field is also in millimetres. Thus, any object on the field that has been detected by the camera can be converted into coordinates on the field relative to the robot. You may also notice a slight inaccuracy between the computed location between the projected coordinates of the ball and the real coordinates of the ball, as shown in the diagram. This could be compensated by using a z-plane offset by 21mm so the coordinates actually lie centered in the ball, but real-life testing found this inaccuracy to be negligible. Math details (click to expand)</summary> Calling cv2.fisheye.undistortPoints</code> without the P</code> keyword parameter yields the coordinates of each point on the image plane (the plane one unit distance away in the direction the camera is pointing in). If we treat the camera point as the origin of our coordinate system, and +Z points up then the ray has parametric form: $$ \vec{P} = t\vec{D} $$ where $\vec{D}$ is the vector given by $\bigl( \begin{smallmatrix} u \\ 1 \\ v \end{smallmatrix} \bigr)$ (if our camera points in the +Y direction, and (u, v)</code> is the normalised cooridnates returned by undistortPoints</code>). As 3 separate equations: $$x = ut$$ $$y = t$$ $$z = vt$$ We want $z = -H$ where $H$ is the height of the camera above the ground in millimetres. So: $$t = \frac{-H}{v}$$ Notice that $t$ must be greater than zero for there to be an intersection. Then we can say: $$\vec{P} = -\frac{H}{D_z}\vec{D}$$ Though since the camera on our robot points below the horizontal, we need to apply a rotation matrix to $\vec{D}$ first. scipy</code> conveniently provides a function to do this for us: from scipy.spatial.transform import Rotation as R rotation = R.from_euler("x", -CAMERA_ELEVATION_ANGLE) ray = rotation.apply(ray) </code></pre> My code initially did this essentially, however there is an optimisation that we can do to make it much more efficient to do these operations in bulk (which was how the projection on the entire image above was possible in less than a second). A homography is a way to perform any perspective transformation between two planes. To understand homographies, we must first define homogenous coordinates</a>. These are coordinates introduce an extra dimension to points: so (x, y) is represented by (x, y, 1). These coordinates are scale-invariant, meaning (2x, 2y, 2) is the same 2D point as (x, y, 1). Homogenous coordinates are useful because they allow more transformations to be represented by matrices compared to cartesian coordinates. A homography is an invertible matrix that operates in this coordinate system. The ray-plane intersection operation can be performed as single 3x3 homography: First obtain the rotation matrix $\mathbf{R}$ to represent the rotation of the camera. Then we want a homography that does: $$x' = -\frac{H}{z} x$$ $$y' = -\frac{H}{z} y$$ $$z' = -H$$ This already looks like an operation on homogenous coordinates because any homogenous coordinate $(x, y, z)$ is converted into the cartesian coordinates $(\frac{x}{z}, \frac{y}{z})$. So we want a matrix that turns $(x, y, z)$ into $(-Hx, -Hy, z)$ for this to match our desired transformation. This matrix is trivially: $$ \begin{bmatrix} -H & 0 & 0 \\ 0 & -H & 0 \\ 0 & 0 & 1 \end{bmatrix} $$ You can check this yourself by reading the matrix as columns left to right: each column describes where its corresponding coordinate is mapped to. After converting the homography into cartesian coordinates, the two coordinates will be coordinates on the plane of the field. So the one homography matrix is: $$ \mathbf{Q} = \begin{bmatrix} -H & 0 & 0 \\ 0 & -H & 0 \\ 0 & 0 & 1 \end{bmatrix}\mathbf{R} $$ This is effectively what the code does, apart from the coordinate system being slightly different (+Y-up instead of +Z-up). You can read it here</a>. </details> Looking at the projection above, you will notice the large amount of garbage in the lower half of the image. This corresponds to the cases where the ray does not intersect with the field, meaning the object is above the horizon line. Mathematically this yields a computed location behind the camera, and the code that projects points individually discards those points. This image was produced by applying the homography matrix directly to the image which is why they are shown. Conveniently, this provides a reliable method to discard ball or goal detections outside the field. Thus, with only a single forward-facing camera, my algorithm can compute relative coordinates of any visible object on the field. In the time we had, I was only able to take advantage of this to better capture the ball, but as can be seen in the above image, the robot can get quite a lot of information about its field location based on this data. This is a classic state estimation problem: sensor data provides incomplete and noisy data, and our goal is to infer the true state of the system. It was always my goal to implement an extended Kalman filter to fuse odometry and vision data to get good state estimates of robot positions and velocity. This would enable true localisation, which would enable far more sophisticated strategies. I studied this free online book</a> to learn about these filters, written by the author of filterpy</code>. All bayesian filters operate around a loop of predict and measurement steps. IMU, odometry and velocity command data would be used during the predict steps to update the robot's state forward in time based on a motion of how these relate to the robot's state variables (position and velocity). Kalman filters keep track of error with a multivariate Gaussian distribution, and the error covariance grows during this step. Then, in the measurement step, data from field lines constitute absolute measurements of a robot's position. Different types of measurements can be made based on the type of 'feature' visible of field lines, for example seeing only two parallel field lines a given distance apart may provide a measurement of only the y and θ axes, while seeing a corner would update x, y and θ. Measurements can also incorporate uncertainties propagated from the vision system: for example a small segment of field line would propagate a greater uncertainty than a larger one. Another useful application of filters that I wanted to implement was ball deduction. If the location of both robots is known, the ball not being visible actually conveys a large amount of information as to the location of the ball. A discrete bayesian filter breaks up the field into a number of rectangular 'cells', and assigns a probability of the ball being in that location. This is initially set to be a uniform probability distribution. Then, if the robot does not observe the ball in a cell, it can scale the probability for that cell down by a large factor (the probability remaining would be nonzero due to the uncertainty of measurement). This is the measure step. When the ball is seen, high probability can be assigned to those regions and a different state estimation technique such as a Kalman filter would be used instead. The update step is applying a convolution</a> to the grid to propagate beliefs forward through time, as when time passes it becomes more uncertain whether those measurements in the past still apply. This algorithm can quickly deduce where the ball is based on the area of the field that is visible (and the projection techniques mentioned earlier give us exactly this data), so the robot can effectively gain a sense of object permanence and infer things about the ball location even when it is not visible. It is a multi-hypothesis state estimation algorithm, making it suitable for this type of deduction as multiple possible locations for the ball can be considered. This also gives rise to more complex defensive strategies. A popular RoboCup Junior strategy seen in international games is 'hiding the ball' where the ball is controlled by a dribbler to face an angle where it is not visible to other robots. This technique would defeat that, which is why we wanted to implement it. However, we did not see this technique in use at our state competition. Technically, the partial localisation data is enough to implement a 'hiding the ball' strategy, but I only achieved partial localisation a few days before the competition, and the ball acquisition improvements to follow were higher value per unit effort. Ball acquisition tactic</a></h2> The next problem is getting the robot to the ball, specifically such that it is within the possession of the dribbler. This is a control theory problem: I had to write some sort of controller that determines the appropriate command velocities for x (longitudinal movement), y (lateral movement) and θ (yaw rotation). In my initial implementation before partial localisation, I implemented it in the simplest way I could think of: If the ball is to the left, turn left</li> If the ball is to the right turn right</li> If the ball is within a central region in the image, go forwards</li> If the ball is not visible spin in circles</li> </ul> The last step could be improved with the discrete Bayesian logic described above, but most of the time the ball is visible to at least one robot. Thus, improving how the robot gets to the ball when it is visible is the most useful part to optimise. Notice that the logic above takes no advantage of the lateral movement capability of the robot, so its path is necessarily more indirect. Thinking about translational movement, the robot should always move in a vector towards the ball. Thus, the translational velocity should always be equal to the relative displacement of the ball (as determined by the projection algorithm described earlier) multiplied by some scalar. We want the robot to be aligned with the ball for it to enter the dribbler, so this is something a PID controller can do. At first glance, the translation also seems like something suitable for PID, however PID does exactly the opposite of what we want to do. PID control tries to get a system's state to a certain value and have it stay at that value, while we want to move towards the ball with some forwards velocity in order for it to be captured by the dribbler. Thus, the displacement vector approach to translation is the better approach. The code for this policy is rather simple: theta = np.arctan2(target[0], target[1]) vel_theta = self.pid_theta(theta) # assume vel_scale is a scalar constant for now velocity = np.array([target[0], target[1]]) * vel_scale(theta) # Clip velocity vector length to 300mm/s max vel_length = np.linalg.norm(velocity) if vel_length != 0: velocity *= min(vel_length, 300) / vel_length # X-forwards coordinates self.command.send_command(velocity[1], velocity[0], vel_theta, 1) </code></pre> See the full code for the chase behaviour here</a>. In the days before the competition, I just hardcoded a constant translational velocity, and this policy was used through most of the competition (I was busy debugging hardware issues). This was generally successful at capturing the ball at long range, but when the robot started out close to the ball but misaligned, it would move towards the ball with not enough time to properly capture it in the dribbler. So in the minutes before a match, I decided to fix it. The robot's speed should be a function of its alignment with the ball, with the speed being maximum when it is perfectly aligned, and zero when it is fully out of alignment (so that the robot has time to turn towards the ball). This reeks of $y = 3\cos\theta$, however I thought this would not be 'aggressive' enough. The dribbler is quite wide, so I wanted to prefer speed over alignment slightly more than what that graph suggested. So I opened up Desmos</a> and as a team we quickly found the parabola through those points. In retrospect I should have fitted a quartic, since this graph is only slightly more aggressive than cosine. This improved the robots' ability to capture the ball at short range. Open source release</a></h1> Mechanical CAD, PCB designs and code are all released under open source licenses in our GitHub repository</a>. I hope this can be of use for future teams.

Writing a Unix-like Operating System: Part 1 2025-09-29T00:00:00+00:00 Scope: To build an operating system kernel that runs on real hardware (my Dell Inspiron 13 5330) resembling an early console-based Unix (such as Sixth Edition Research Unix) in kernel functionality. This post documents my progress in building a Unix-like operating system for the x86-64 platform. It contains a brief overview of the state at the time of writing, followed by more technical discussion of the design decisions I've made. The source code for this operating system is available here</a>. This OS is still a work in progress, and is missing some key Unix features: Signals</li> Persistent filesystem</li> Working directory and relative paths in system calls</li> Environment variables</li> Many system calls pertaining to creation and deletion of files and directories</li> </ul> This kernel is currently highly unstable, and tends to crash. There is much work to be done on error handling. I plan to write about these features in a Part 2, once I have implemented them. Overview</a></h1> To learn more about modern operating systems, I've built an OS kernel for the x86-64 platform (also known as AMD64). It is written in the Rust programming language, and runs on bare-metal. Current features: A simple userspace shell</li> A kernel-mode text console device (controlled by reading and writing to /dev/console</code>)</li> A rudimentary VFS (Virtual file system) supporting different file systems (including a device file system which contains /dev/console</code>) under a single root file system</li> A subset of Linux syscalls following Linux syscall calling conventions</li> Loading of static 64-bit ELF executables.</li> Preemptive multitasking on a single processor</li> fork()</code> and exec()</code> based process spawning</li> Drivers for the GOP framebuffer (the graphical interface provided by the UEFI firmware), the APIC interrupt controller, and for the emulated PS2 keyboard connected through the IOAPIC</li> </ul> A detailed explanation of these features will follow. References</a></h2> OSDev Wiki</a></li> Intel® 64 and IA-32 Architectures Software Developer's Manual</a></li> Linux syscall reference | x64.syscall.sh</a></li> Linux man pages (eg: man 2 read</code>)</li> Philipp Oppermann's blog | Writing an OS in Rust</a></li> Wesley Aptekar-Cassels | How to set up the APIC to get keyboard interrupts</a></li> System V ABI</a></li> mit-pdos/xv6-public: xv6 OS</a></li> redox-os / kernel</a></li> Linux kernel UAPI headers</a></li> Operating Systems: Three Easy Pieces</a></li> Rust-OS Kernel - To userspace and back! - nikofil’s blog</a></li> Malloc tutorial</a></li> </ul> Previous attempts</a></h1> This is not my first attempt at building an OS. Soon after learning Rust in December 2022, I came across Philipp Oppermann's excellent Rust OS tutorial series</a>, which was my initial exposure to OS development. The series provides a thorough walkthrough of the initial setup process of creating an OS for x86-64 (including text output, interrupts, paging and heap allocation). This lead me to several attempts at creating an operating system: bench</a>: An x86-64 C kernel using Limine bootloader (BIOS-only) In this project I applied what I had learned from the Rust tutorials in a different context</li> Most progress here was with understanding interrupts with the IDT (Interrupt Descriptor Table) and the legacy 8259 Programmable Interrupt Controller</li> </ul> </li> bench2</a>: An x86-64 Rust kernel using the bootloader</code> crate with support for UEFI booting. In this project, I mostly followed the Rust tutorials for implementing core structures (IDT, GDT, page table, heap), but focused on getting keyboard support on UEFI. At the time of this writing, the Rust OS tutorials rely on some functionality only available when booting from the legacy BIOS (8259 PIC and VGA text mode), which now have more complex replacements in UEFI. In this attempt, I implemented keyboard input by following overall steps from this post</a>.</li> </ul> </li> </ul> Why build a Unix clone?</a></h1> I started benchix (this project) because I found myself getting stuck on architectural decisions in bench2. As a beginner to OS development, I wanted to focus on the mechanisms of operating systems rather than the policies. This led me to re-scope the project to specifically implement a Unix-like kernel. In benchix, this means a few things: A virtual file system allowing devices to be accessed as files</li> Unix-like system calls (read</code>, write</code>, open</code>, close</code>, brk</code>, fork</code>, execve</code>, etc.)</li> </ul> However, benchix isn't a direct port or rewrite of any particular kernel. Where possible, I based my implementation on descriptions of kernel features from man pages or OSTEP (Operating Systems: Three Easy Pieces). The kernels I did reference were xv6 and Redox (linked to in references). I mention areas where my implementation is based on one of these kernels. Implementation details</a></h1> This is a roughly chronological record of my implementation, grouped by feature. The bootloader</a></h2> The Rust bootloader</a> crate provides a way to create a bootable disk image from a kernel. They have a guide</a> on setting up the two crates needed to boot. The bootloader provides an entry_point!</code> macro for use in the kernel and will pass a boot_info</code> struct to the annotated function. This contains some important information such as: The offset that physical memory is mapped at (memory mappings discussed later)</li> A pointer to the GOP framebuffer</li> The physical memory layout (used to determine what physical memory frames are available for allocation)</li> A pointer to the RSDP. The RSDP is the root pointer to the boot-time ACPI (Advanced Configuration and Power Interface) tables. They are very verbose so I use the acpi</code> crate to parse these structures. I need this information to set up the interrupt controllers to receive keypresses.</li> The address the ramdisk is mapped at</li> </ul> The other part of the bootloader crate runs in a build.rs</code> at the root of the workspace. This file builds the C userspace into a .tar</code> archive, and uses Cargo's binary dependency feature to get the kernel binary. Then, it invokes the bootloader crate to create a GPT (GUID Partition Table) disk image. Framebuffer and panic handling</a></h2> I use the noto_sans_mono_bitmap</a> crate to display monospaced text on the framebuffer. I have a separate console (PanicConsole</code>) for kernel panic that does not rely on heap allocated memory. My main console uses a heap-allocated backing text-buffer to hold text to avoid having to read from frame-buffer memory when scrolling (which can be slower). The panic console uses a static mut</code> variable storing a pointer to the buffer and re-initialises the display from there. This is necessary because if the framebuffer was stored behind a lock, the panic handler may deadlock if the panicking thread was holding that lock. After the heap is available, my main console is initialised from the framebuffer, and it is owned by the boot thread until the initialisation of the VFS. After this it is passed to the device filesystem to be accessed at /dev/console</code>. IDT and GDT</a></h2> Initially, I was able to bring over my IDT (interrupt descriptor table) and GDT setup from my old bench2 kernel (which itself matches closely with the Rust OS tutorials). To manage these structures I use the x86_64</code> crate, which contains useful wrappers around Intel 64 registers, structures and instructions. In the original bench</code> kernel (in C), I implemented some of these by hand, but delegating these operations to a crate lets me narrow my scope to focusing more on the specifics of building a kernel (threading, system calls, scheduling, concurrency) rather than spending time on Intel 64 peculiarities. I did have to make some modifications for userspace: Adding "flat" code and data segment descriptors for userspace (these are mandatory but since I am in Long Mode they don't actually set any segmentation limits).</li> The TSS (task state segment, which used to hold the task state in 32-bit mode, but in 64-bit mode now holds the interrupt stack table) gets a new entry (RSP0) to describe the stack to use when an interrupt occurs in Ring 3 (since interrupts are handled in Ring 0). In this case I use the current kernel stack for that thread. If I had one global stack to use for interrupts, then that stack would get overwritten if I do a context switch and then another interrupt occurs. This would mean that the execution would not be able to return to the previous thread, since the return address on the stack would have been overwritten.</li> </ul> Interestingly, long mode supports segmentation for the FS and GS segments. Most kernels use this to store per-processor information (like the current kernel stack). I don't have multiprocessing yet so I have just been using what is effectively a static mut</code> to store this information. However, I will consider using this approach when I add multiprocessing. Memory management</a></h2> Bootloader mappings</a></h3> Vladsinger</a>, CC BY-SA 3.0</a>, via Wikimedia Commons My kernel is a higher-half kernel. On x86-64, there are only 48 usable bits of the virtual address, and the rest are a sign-extension. This essentially breaks the address space into two "halves", which can be distinguished between eachother using only a single bit check. Since virtual address space is so large (256 TiB!), I use the entire higher half for my kernel's memory. This allows syscall handlers to quickly check whether a given buffer belongs to the process, to avoid the process reading the kernel's address space (however, I do not currently mitigate against speculative execution exploits). The bootloader</code> crate supports configuring the addresses mappings it creates. This saves a bit of work configuring the address-space manually (and the physical offset mappings are needed to actually access the page tables). pub const HEAP_START: u64 = 0x_ffff_9000_0000_0000; pub const KERNEL_STACK_START: u64 = 0xffff_f700_0000_0000; pub const KERNEL_STACK_SIZE: u64 = 80 * 1024; // 80 KiB pub const LAPIC_START_VIRT: u64 = 0xffff_8fff_ffff_0000; pub const IOAPIC_START_VIRT: u64 = 0xffff_a000_0000_0000; pub static BOOTLOADER_CONFIG: BootloaderConfig = { let mut config = BootloaderConfig::new_default(); config.kernel_stack_size = KERNEL_STACK_SIZE; config.mappings.kernel_stack = Mapping::FixedAddress(KERNEL_STACK_START); config.mappings.physical_memory = Some(Mapping::FixedAddress(0xffff_e000_0000_0000)); config.mappings.dynamic_range_start = Some(0xffff_8000_0000_0000); config.mappings.dynamic_range_end = Some(0xffff_8fff_fffe_ffff); config }; </code></pre> This configuration ensures that all kernel mappings created by the bootloader are in the higher-half. The bootloader is configured to map the entire physical address space at a fixed offset (0xffff_e000_0000_0000</code>) to allow us to access the page-tables (which store physical addresses). I have separate mappings for IOAPIC_START_VIRT</code> and LAPIC_START_VIRT</code> which are used elsewhere to create no-cache mappings for these memory-mapped interfaces. The dynamic range gives the bootloader a range to place mappings which I haven't specified (for example the ramdisk). Physical Memory Management</a></h3> The physical memory manager allocates frames of physical memory for the kernel to use. These frames can be then mapped to virtual memory, or used through the physical offset mappings. Apart from the mappings allocated by the bootloader, all mappings are created by allocating one of these frames and creating a virtual address mappings. One feature my allocator includes that wasn't in the OS tutorials is deallocation. This allows physical frames to be reused after they are done with. This requires the memory manager to keep track of free memory separately from the structures provided by the bootloader. I use a bitmap structure for this: #[derive(Debug)] pub struct PhysicalMemoryManager<'a> { bitmap: &'a mut [u64], // 0 for free, 1 for used physical_offset: VirtAddr, } </code></pre> Each 0 or 1 in the bitmap corresponds to the availability of a 4KiB physical frame, starting at physical address 0 up to the highest physical address. Creating this structure involves the following process: Find the highest address in the bootloader mappings to determine the bitmap size.</li> Find the first available memory region in the bootloader mappings that is large enough to store such a bitmap.</li> Get a &'a mut [u64]</code> reference to that region (using the physical offset to get a virtual address to store in the base pointer) and store that as our bitmap.</li> Mark any reserved regions as used (the bootloader specifies this) as well as marking the frames backing the bitmap as used.</li> </ol> Now, frames can be allocated quite simply: unsafe impl<'a> FrameAllocator<Size4KiB> for PhysicalMemoryManager<'a> { fn allocate_frame(&mut self) -> Option<PhysFrame<Size4KiB>> { for (idx, entry) in self.bitmap.iter().enumerate() { if *entry != u64::MAX { let frame = PhysFrame::containing_address(PhysAddr::new( (idx as u64 * 64 + entry.trailing_ones() as u64) * 4096, )); self.set_frame(frame); return Some(frame); } } None } } </code></pre> The advantage of the bitmap approach is its simplicity and memory efficiency. If I wanted faster allocation without searching through the bitmap every time, I could have a frame stack and allocate in bulk, but this is simple enough for now. Page table management</a></h3> Here I use the OffsetPageTable</code> from the x86_64</code> crate to create the actual page table entries to map between a specific page and frame. The OffsetPageTable</code> stores an offset to physical memory, which allows it to access the page table entries through virtual memory. Kernel Heap</a></h3> This is another place where I used an external crate. I use the linked_list_allocator</code> crate to implement a GlobalAlloc</a>. This allows me to use Rust's alloc</code> types like Vec</code> and Box</code>. These types do have the disadvantage of having kernel panics when allocation fails. Virtual File System</a></h2> My VFS is informed by James Molloy's Tutorial</a> (archived link)</a></li> </ul> The virtual file system (VFS) allows for multiple different file systems to be mounted together in a single file system. My VFS only supports mount points as direct children of the root directory /</code>. This is sufficient to have the two mappings I currently use: The device FS mounted at /dev</code></li> The ramdisk mounted at /bin</code></li> </ul> Of particular interest is the VFS inode struct: pub enum FileType { File, Directory, Device, // Just block devices for now, we don't have a good distinction between buffered/unbuffered devices } pub struct Inode { pub dev: u32, // File system that the file belongs to pub inode: u32, // inode number pub file_type: FileType, pub size: usize, // 0 for devices pub major: Option<u32>, // Device driver pub minor: Option<u32>, // Specific device that belongs to driver pub inner: Option<Box<dyn Any + Send + Sync>>, } </code></pre> The (dev, inode)</code> tuple uniquely identifies an inode on the file system. The major</code> and minor</code> numbers are used by the device filesystem to identify particular devices, it isn't actually necessary with a VFS but it mirrors the way xv6 does it. These inodes are the VFS' view of the actual inodes which the drivers retrieve. All of my current filesystems actually store these inodes in memory, but a real persistent file system would retrieve these inodes from the disk. The information stored inside the inode inner</code> field (in persistent filesystems these are usually references to different blocks that form the file) are an implementation detail of the specific driver. pub trait Filesystem: Send + Sync { fn open(&self, inode: Arc<Inode>) -> Result<(), FilesystemError>; // Usually increments a reference counter or starts caching inodes -- doesn't lock the file fn close(&self, inode: Arc<Inode>) -> Result<(), FilesystemError>; // Decrements the reference counter, or removes from cache fn read( &self, inode: Arc<Inode>, offset: u64, buffer: &mut [u8], ) -> Result<usize, FilesystemError>; // A locking operation fn write( &self, inode: Arc<Inode>, offset: u64, buffer: &[u8], ) -> Result<usize, FilesystemError>; // A locking operation fn readdir(&self, inode: Arc<Inode>) -> Result<Vec<DirectoryEntry>, FilesystemError>; // Get directory entries fn inode(&self, dev: u32, inode: u32) -> Result<Arc<Inode>, FilesystemError>; // An inode lookup fn traverse_fs(&self, root: Arc<Inode>, path: &str) -> Result<Arc<Inode>, FilesystemError> { ... // default implementation omitted for brevity } } </code></pre> The inode()</code> interface allows for retrieving of inodes based on the (dev, inode)</code> pair. It returns an Arc<Inode></code> which is reference counted. This is useful for persistent file systems where the inodes are not already in memory, because they can be held in a cache and dropped when the reference count goes to 0. The VirtualFileSystem</code> (which itself implements this trait), allows for mounting of these filesystems. It is initialised as follows: VFS.init_once(|| { let mut vfs = VirtualFileSystem::new(); let devfs = Devfs::init(console, 1); let ramdisk = unsafe { Ramdisk::from_tar(2, &binary) }; vfs.mount(1, Box::new(devfs), "dev", 0).unwrap(); vfs.mount(2, Box::new(ramdisk), "bin", 0).unwrap(); vfs }); </code></pre> This initialises the /bin</code> filesystem from the uncompressed tar ramdisk loaded into memory by the bootloader, and the /dev</code> filesystem which currently only holds /dev/console</code> (implementing console writes on write, and keyboard input on reads). Userspace</a></h2> Relevant reference: Rust-OS Kernel - To userspace and back! - nikofil’s blog</a> I use the fast system call instructions: SYSCALL</code> and SYSRET</code> to switch between Ring 0 (kernel mode) and Ring 3 (user mode). Some setup is required before these instructions can be used: Enabling system call extensions</li> Writing the syscall handler address to the LStar MSR (model‑specific register)</li> Specifying that interrupts should be disabled on entry to the system call handler (using the SFMask MSR)</li> </ol> unsafe { Efer::write(Efer::read() | EferFlags::SYSTEM_CALL_EXTENSIONS); } LStar::write(VirtAddr::from_ptr(handle_syscall as *const ())); SFMask::write(RFlags::INTERRUPT_FLAG); </code></pre> Once this is done, I can enter user mode using a single sysret</code> instruction. This instruction uses the rcx</code> register to store the userspace entry point and the r11</code> register to hold the new flags upon entering. A specially crafted thread entry point to enter into user space will be covered later. To use this, I need to load a user space executable. My kernel loads static ELF64 executables, and the loader implementation is almost entirely based on the format's description on Wikipedia</a>. Loading a static executable involves the following steps: Validating the ELF header (checking the magic and executable type matches is expected)</li> Using the ELF header to locate the program headers.</li> Use the information in the program headers to load the program segments into memory.</li> </ol> The program headers contain information to load the actual program segments in the executable. A program header entry looks like this: #[repr(C)] pub(super) struct ProgramHeaderEntry { pub(super) segment_type: u64, // contains both p_type and p_flags pub(super) offset: u64, pub(super) virtual_address: u64, pub(super) unused: u64, pub(super) image_size: u64, pub(super) mem_size: u64, pub(super) align: u64, } </code></pre> The segment_type</code> field actually contains to 32-bit fields inside: let segment_type = header.segment_type as u32; let segment_flags = (header.segment_type >> 32) as u32; </code></pre> We filter for headers that have segment_type == 1</code> (P_LOAD) as they indicate segments to actually load into memory. let executable = (segment_flags & 1) > 0; let writable = (segment_flags & 2) > 0; let readable = (segment_flags & 4) > 0; </code></pre> The segment flags tell us whether the segment is executable</code>, writable</code> and readable</code>, which can be applied to the actual memory mappings. To create these mappings, I allocate new physical frames, and then map these to the pages behind the address range virtual_address..virtual_address+mem_size</code>. Note that mem_size</code> can differ from image_size</code>, which is the size of the data to copy. This is used by the exectuable for the .bss</code> section (storing uninitialised variables), to inform the loader to reserve space for these variables but not actually copy any data into it. Next, I allocate the stack. The contents of the stack on program initialisation is defined by the System V ABI</a> in Section 3.4.1 Initial Stack and Register State</code>. This involves allocating: An information block (containing argument strings, environment strings)</li> Auxiliary variables (used by libc for things such as TLS (thread local storage), however I removed any auxiliary variables after not having any success with setting this up)</li> Pointers to environment variables (null since I don't support environment variables yet)</li> Finally, argc</code> and the argument pointers: The argument pointers point to the argument strings in the information block, since the SysV ABI expects the argument array to actually be contiguous</li> </ul> </li> </ul> The userspace entry point (_start</code>) actually retrieves these values from the stack, and passes them to main</code> as int argc</code> and char** argv</code>. The char** argv</code> actually points to the contiguous array of argument pointers on to the stack, and each char*</code> entry points to a null-terminated string in the information block. System calls</a></h2> System call handler</a></h3> I'll now describe the handle_syscall</code> function from earlier. This uses Rust's naked functions</a> feature to create a bare assembly handler which doesn't include the normal function prologue and epilogue generated by the compiler. What's important to note is that the syscall</code> instruction doesn't change the current stack or save any registers from the calling function. Unlike interrupt handlers, the CPU doesn't do any of this automatically. Instead, the stack switching has to happen manually. To understand the desired semantics of a system call, the System V ABI</a> can be consulted. In Section A.2.1, the system call interface for Linux is listed as an informative appendix. My system call interface is very similar except that my syscalls are limited to 4 arguments instead of 6. At this point, it’s important to understand the semantics of a sysv64</code> function call. These are the callee-preserved registers for the System V ABI: rflags</code></li> rbx</code></li> r12</code></li> r13</code></li> r14</code></li> r15</code></li> rbp</code></li> rsp</code> What this means is that upon the function return, the contents of these registers should be the same as they were when called. Any other registers can be freely modified, and if the caller needs to keep the values, they should be saved before calling the function.</li> </ul> My system call interface follows these semantics for preserving of registers. This differs from the Linux kernel, which preserves all registers except rcx</code>, r11</code> and rax</code> (at least according to this non-normative section of the ABI). The main part of the system call handler is as follows (split up and as excerpts): The syscall handler saves rip</code> into rcx</code>, and rflags</code> into r11</code>. These values are used by sysret</code> to return to userspace, so they are saved to the stack. I also save rbp</code> and rbx</code>, which will be used soon.</li> </ol> push rcx push r11 push rbp ; will store old sp push rbx ; will store new sp </code></pre> To switch stacks, I need to get the current thread's kernel stack. Currently, this involves a separate function call to some Rust code which gets it from a PerCPU</code> struct (which is itself a wrapper around an UnsafeCell</code> because I don't have multiple processor support yet). I wrote this code before I knew about FSGSBASE</code> which allows code to set the base of the fs</code> and gs</code> segments. Since these are per-processor, they can be used to store information such as the current kernel stack to be accessed without a function call. But because I do use a function call, I need to backup the syscall arguments, since they are in caller-saved registers. After the function call, I use rbx</code> to store the new stack, allowing for the syscall params to be restored.</li> </ol> push rax ; syscall number push rdi ; arg0 push rsi ; arg1 push rdx ; arg2 push r10 ; arg3 call get_kernel_stack ; Return value is now in rax mov rbx, rax ; RBX = new sp ; Restore syscall params pop r10 pop rdx pop rsi pop rdi pop rax </code></pre> Switch kernel stacks.</li> </ol> mov rbp, rsp ; backup userspace stack mov rsp, rbx ; switch to new stack </code></pre> Call an inner function to handle the system call in Rust. To do so, reshuffle the arguments to follow the sysv64 calling convention.</li> </ol> push rax ; syscall number push rdi ; arg0 push rsi ; arg1 push rdx ; arg2 push r10 ; arg3 ; Pop to follow normal sysv64 calling convention pop r8 pop rcx pop rdx pop rsi pop rdi call handle_syscall_inner </code></pre> Finally, return from the system call. The return value is now in rax</code> from the previous function.</li> </ol> mov rsp, rbp ; Restore userspace stack pop rbx pop rbp pop r11 pop rcx sysretq </code></pre> System call implementation</a></h3> From here, most system calls are not particularly complicated. I've implemented check_addr</code> and check_buffer</code> utilities to ensure that pointers and buffers passed to system calls actually point to lower‑half memory. However, I don't have any validation past this, so a process can easily crash the kernel by passing it an invalid pointer. I've implemented these system calls: read</code></li> write</code></li> open</code></li> close</code> (a no-op for now)</li> exit</code></li> arch_prctl</code> (I used this when trying to integrate the musl libc. I'm using my own pseudo-libc for now, since that was taking too long to integrate.)</li> execve</code></li> brk</code></li> fork</code></li> waitid</code></li> </ul> Kernel Multithreading</a></h2> Kernel multithreading is the prerequisite which allows for preemptive multitasking of userspace threads. To be able to switch between threads on a timer interrupt, it is far easier to just swap out the kernel context in a fairly controlled way compared to having one kernel thread somehow returning to different userspace stacks. Almost all kernels use this approach of having one kernel-stack per userspace thread. Context switching</a></h3> Once a thread is in kernel mode, the userspace stack and saved registers live at the bottom of the kernel stack (placed there by the system call handler to return to later). This means that to switch between threads, only the kernel stack needs to be swapped out. Take another look at the callee-preserved registers: rflags</code></li> rbx</code></li> r12</code></li> r13</code></li> r14</code></li> r15</code></li> rbp</code></li> rsp</code></li> </ul> To actually execute ret</code> from a function and continue executing the rest of the thread, these are the only registers which need to be saved. Note that rip</code> is not on that stack, since the processor will jump back to the return address that is on the stack. Thus, a "context switch" involves saving and swapping out these values to switch between threads. I found two approaches to this. xv6's approach</a> pushes all of these values (except esp</code>) to the stack, and then using the stack pointer as the only thing that needs to be swapped when context switching</li> Redox's approach</a> uses a struct containing all of the callee-saved registers, and then directly swaps from this struct when context switching</li> </ul> xv6's approach is elegant because it stores context in a single value (esp</code> which is the 32-bit equivalent of rsp</code>). However, to create a new context, a stack must be constructed manually. I didn't like this, so I chose Redox's approach which allows for easy changing of these registers. Being able to change these registers directly also proved useful for both fork()</code> and exec()</code>. For the actual assembly to perform this swapping of contexts, I use Redox's x86-64 switch_to</code> handler</a> (with some modifications). Scheduler</a></h3> To decide what threads to run, I wrote a round-robin scheduler. My scheduler has a main queue called READY</code>, where Arc<Mutex<Thread>></code>s get added to. The scheduler picks the next thread to switch to by popping the front element on this queue. If there are no such elements, then it switches to an idle thread (which halt loops). The arrangement of locks in here is also interesting. As well as a Mutex</code> on each of the threads, there is a global mutex surrounding the READY</code> queue. Since queue operations may happen inside an interrupt handler (for example the pre-emptive multitasking caused by the timer interrupt), interrupts must be disabled when handling this queue to avoid deadlock. After the context switch operation, I must manually release the lock on the previous thread in switch_finish_hook</code> (Rust code called by context switch). This code also sets the page-table mappings for the new thread, and sets the IST such that interrupts will be handled by the current kernel thread if they occur in user-mode. One thing to note is that my yield_execution</code> does not return the current thread to the queue. This is useful for cases where the thread has to wait for something, where the thread itself can be pushed to a different queue, and returned to the READY</code> queue when the operation is done. This is used for both keyboard input and waitid</code>. For preemptive yields on a ready thread, a wrapper yield_and_continue</code> can be used to requeue the current thread to READY</code>, and then yield. execve()</code></a></h2> For the most part, execve</code> is a system call wrapper around the behaviour I already built for ELF64 loading. Since execve</code> overrides previous mappings, I added code to clear the entire lower half of page-table mappings before creating any new ones. Additionally, to reset the thread to begin at the desired entry point, I use a neat hack. The execve</code> code modifies the context struct itself (which is just a struct of callee-saved registers), to store the userspace entry point and initial stack pointer. Then, I set the kernel stack (which is itself allocated on the kernel heap), to have a single element (a pointer to the enter_userspace</code> trampoline). Then, when the context switch calls ret</code>, it will return to the trampoline as if it was called from there. Once in the trampoline, the user space stack and entry point can be moved into the appropriate registers for sysret</code>, now returning from the system call in a completely different process. fork()</code></a></h2> This system call was quite challenging to implement. It would have been easier just to spawn a fresh process (like some modern kernels), but I wanted to understand how it works. Attempt 1: Cloning the kernel stack</a></h3> Initially, I thought I'd implement this by cloning a thread struct, but changing the parent process to point to a forked process. The general steps for this would be: parent_pid = get_pid() yield_execution() if get_pid() == parent_pid() { // Fork hasn't happened yet child_pid = fork_process() child_thread = clone_current_thread() // this would clone from the last yield point, so child execution would start from there set_thread(child_pid, child_thread) enqueue(child_thread) return child_pid } else { return 0 } </code></pre> I started implementing this, as it would avoid having to write more assembly to create a custom return for fork()</code>. However, I soon realised that I would have to clone the kernel stack to do to this. This is problematic as I have the kernel stack allocated on the heap (which is itself problematic for different reasons), which is shared between all kernel threads. Cloning the kernel stack seemed much more complicated then just writing the assembly returns, so I decided to go for that approach. Attempt 2</a></h3> First, we fork the page table. This involves recursively (4 levels) cloning any pages that were user‑accessible and present. Any kernel mappings could be safely shared. Then, I assign the frame of the level 4 page table to this process, so that it can be loaded into the cr3</code> register by switch_finish_hook</code>. Now that the new process has been created, the parent thread can easily return the child PID. However, the child process needs to somehow return to the same place, but returning 0. Kernels like xv6 do this by assigning to the trapframe (registers saved by the CPU when entering the interrupt handler). However, in x86-64, syscalls don't have a trapframe! I couldn't find any information on implementing fork()</code> with the sysret</code> instructions x86-64. To solve this, the information needed to return to userspace can be pushed on the stack manually. The base of the kernel stack is stored in the Thread</code> struct, and since the kernel stack is empty at the start of a syscall, this can be easily located. push rbx push r12 push r13 push r14 push r15 push rbp call handle_syscall_inner </code></pre> Note the omission of rsp</code> here, since the kernel stack is changing anyway. The purpose of preserving these registers is that some important information (eg: location of old stack and some callee-preserved registers from user space) are stored there. let trapframe = current_thread.kstack.last_chunk::<6>().unwrap(); // What we pushed last will have the lowest address (first in our slice) thread.context.rbp = trapframe[0]; thread.context.r15 = trapframe[1]; thread.context.r14 = trapframe[2]; thread.context.r13 = trapframe[3]; thread.context.r12 = trapframe[4]; thread.context.rbx = trapframe[5]; </code></pre> These 6 registers can be transferred into the new thread using the Context</code> struct. Then, the fork trampoline simply clears the rax</code> (return value) register and jumps to the normal syscall return sequence. How to save users from their own network connection 2023-12-20T00:00:00+00:00 The dilemma</a></h2> Consider you have an app. It could be a web app, a native mobile app, or a desktop app. If you have a web app, we are going to assume the app is already client-side rendered and does not need to load from the server every time it is opened, as it is cached in a service worker. I talk about this kind of web app and its tradeoffs more in this article</a>. Setting that aside, whenever you have some data to store or some computation to do you have a critical decision: Server-side or client-side </blockquote> This decision is not just for web apps, but also for mobile apps and desktop apps, as they also might need to use cloud services. In this article, I justify a particular rule of thumb to use for making these decisions, the exceptions to this rule, and some general development tips which fit well with this rule. Do things on the client-side if you can, and on the server-side if you must </blockquote> The network connection</a></h2> When you fetch data over the network, you are passing over a gigantic void of unpredictability, which you cannot control. Your users' devices are no longer tethered to a relatively stable wired connection. Many users will be using WiFi or cellular connections, and sometimes these connections can grow slow, congested or drop out completely. Users no longer have the patience to wait for slow network connections; we all know the three-second statistic</a>. The solution is to do everything that can be done on the client-side on the client. Here are some examples: Usage</th> Client/Server</th></tr></thead> Settings for a single device</td> Client, only needed on that device</td></tr> Social Media post</td> Server, needs to be stored centrally so many users can access it</td></tr> QR code scan</td> Client, most devices are powerful enough to run QR code scanning code without needing cloud support</td></tr> Access control</td> Server, the client side is not a trusted environment and therefore cannot perform access control in a trustworthy way</td></tr> Running an LLM</td> ...</td></tr> </tbody></table> When a network connection is faster than doing the computation on the client</a></h3> Consider the last example in the table. With our rule, we would run it on the client, as technically it can be run on the client, but on most devices it would be far faster to run it on a server and wait for the dreaded network connection. If, for a significant proportion of your users, it is faster to do a computation on the server than on the client, then its best to do it on the server if possible. Another approach if this proportion is not your entire user population, is to start the operation on the client and server in parallel, and when one finishes abort the other. How to make the network more bearable</a></h2> Sometimes you have to fetch across the network, like accessing a piece of shared data or something that needs access control. When this happens, we can still make some optimisations to make it feel more user-friendly. Here are some things you can do to improve your user experience for fetching server-side data: Optimistic updates After making a change to server side state, update it on the client side assuming it succeeded and if it fails then revert it.</li> </ul> </li> Persistent caching Cache server-side data between page loads, so data is shown immediately (state data is better than no data)</li> </ul> </li> Periodic sync When offline, queue up changes to be made to the server-side and sync this in the background when back online</li> </ul> </li> Validation Validate data to avoid undefined behaviour when invalid data is given, either because of an actual bug or a proxy (like on school or corporate networks)</li> </ul> </li> Graceful error handling Keep showing stale data even when an error occurs so the user can work with the stale data until the issues stop</li> </ul> </li> Exponential backoff on retries If an error is temporary it might resolve itself after some period of time, so keep trying a certain number of times. Use exponential backoff to avoid putting unnecessary load on the server.</li> </ul> </li> </ul> This is not easy to implement yourself. You may want to use a data-fetching library, sometimes called an async state manager. This library will be responsible for syncing server-side state with a client-side "store", which your app can deal with, so you don't need to worry about the network as much. I personally use TanStack Query</a> for web apps, and I've heard good things about SWR</a>. I don't currently have any recommendations for native. Also for things like periodic sync and persistence you may need to do some extra configuration. For validation of network requests on the Web, I use zod</a>. How to use OAuth PKCE in client-side JS 2023-05-22T00:00:00+00:00 You need to use some API that requires authentication. It doesn't have an SDK. 2 years ago, I was in this position, I started building Timetabl</a>. The API I was using uses OAuth</a>, a protocol for authorising third-party applications, to access a user's data. I started digging through articles about OAuth. Looking at the available libraries for client side OAuth, I didn't seem to have many options. Most JS client libraries were either not actively maintained, didn't support PKCE (Proof Key for Code Exchange), didn't run in the browser, or were too complicated for me to understand. If you don't understand much about OAuth, or just want a refresher, check out this article</a> for an explainer. What not to do</a></h2> Stubborn, I dug into the RFCs for several weeks, and tried to wrap my head around the complexities of OAuth. In around a month I got a basic prototype implementation working. As my application grew, I started to realise that ~90% of my bugs came from that OAuth implementation. As the complexity increased it became almost impossible to maintain. Eventually, I decided to start looking for OAuth libraries again. A library that makes OAuth easy</a></h2> After some research, I found badgateway/oauth2-client</a>. It is an excellent library that is actively maintained. I'm surprised it dosen't have a bigger community, so give it a ⭐ and contribute if you can. This library has unit tests, and is fairly well-documented. Registering your application</a></h3> Go to the dashboard of your OAuth provider, and create a new application. You will see a page like this: You can ignore the "Client Secret" field, as we will be using PKCE</a> to authenticate. Note down the "Client ID" and "Redirect URI" fields as we will need them for later. Setting up the library</a></h3> Install the library with npm: npm i @badgateway/oauth2-client </code></pre> Next, setup the client class: import { OAuth2Client, generateCodeVerifier } from "@badgateway/oauth2-client"; // Replace the following values with your own const client = new OAuth2Client({ server: "https://my-auth-server/", clientId: "...", tokenEndpoint: "/api/token", authorizationEndpoint: "/api/authorize", }); </code></pre> The values for server</code>, tokenEndpoint</code>, and authorizationEndpoint</code> will be in your OAuth provider's documentation. The clientId</code> is the "Client ID" you noted down earlier. Logging in</a></h3> Add a login button to your page, and add the following code to the click handler: const codeVerifier = await generateCodeVerifier(); // You should really dynamically generate a random string, but this is fine for a demo const state = "some-string"; document.location = await client.authorizationCode.getAuthorizeUri({ // The redirect uri you noted down earlier redirectUri: "https://my-app.example/", state, codeVerifier, // see your OAuth provider's documentation for available scopes scope: ["scope1", "scope2"], }); // Save the code verifier and state to the localStorage localStorage.setItem("codeVerifier", codeVerifier); localStorage.setItem("state", state); </code></pre> This code redirects the user to the OAuth provider's login page. After the user logs in, they will be redirected back to your application. This code will extract the code</code> and state</code> from the URL, and exchange the code</code> for an access token. // Get query string const query = Object.fromEntries( new URLSearchParams(window.location.search).entries() ); // Error check if (query.error) { // Handle error // ... // Logout localStorage.setItem("loggedIn", false); } // If the server returned an authorization code, attempt to exchange it for an access token if (query.code) { const codeVerifier = localStorage.getItem("codeVerifier"); const state = localStorage.getItem("state"); const oauth2Token = await client.authorizationCode.getTokenFromCodeRedirect( document.location as any, { /** * The redirect URI is not actually used for any redirects, but MUST be the * same as what you passed earlier to "authorizationCode" */ redirectUri: config.redirect_uri, state, codeVerifier, } ); // Save the token object to localStorage localStorage.setItem("token", JSON.stringify(oauth2Token)); } // Clear query string window.history.replaceState({}, null, location.pathname); </code></pre> Logging out</a></h3> To log out, simply remove the token from localStorage and set loggedIn</code> to false</code>: localStorage.removeItem("token"); localStorage.setItem("loggedIn", false); </code></pre> Making requests</a></h3> To do this we need to setup the FetchWrapper</code>: const fetchWrapper = new OAuth2Fetch({ client, getNewToken: () => { // This method will only be called when refreshing the tokens fails // You should log the user in again here // `return null` if you want to fail this step }, storeToken: (token) => { // This method will be called when the token is refreshed // You should save the token to localStorage here localStorage.setItem("token", JSON.stringify(token)); }, }, getStoredToken: () => { // This method will be called to get the token from storage return JSON.parse(localStorage.getItem("token")); }, }); </code></pre> To send a request, simply call fetchWrapper.fetch</code> which behaves similarly to plain old fetch</code>: const response = await fetchWrapper.fetch("https://my-api.example/"); // Obviously do some error handling too... const data = response.json(); </code></pre> This FetchWrapper</code> will automatically refresh the token when it expires, and retry the request. It will also handle sending the access tokens. Is it secure to store the token in localStorage?</a></h2> The main issue with localStorage is that it can be accessed by JS. In modern applications, you will often have thousands of dependencies, and if any of them are compromised then you could end up with an XSS attack. You can partially mitigate this risk with a strong Content Security Policy. The JS supply chain is quite resilient to compromised packages, with them being taken down quickly, and the community being notified. It is also very hard to pull off such a targeted attack, through a compromised package. However, let's assume that your app is already compromised through XSS. Even if the token was not stored in localStorage, there would be some other mechanism to make API requests (possibly through a proxy server which stores tokens as HTTP-only cookies), and the attacker could still use that by sending requests from the context of your app. In our case, the attacker will be able to access the token from localStorage</code>, and possibly use it even when the user does not have the app open. However, there is a measure against this called refresh token rotation. Refresh Token Rotation</a></h3> Refresh token rotation ensures that only one party is using the tokens at one time. This prevents attackers from stealing the token and using it outside the application. It goes hand in hand with short-lived access tokens. Short-lived access tokens</a></h4> Access tokens should be short-lived, and expire after a short period of time. This could be anywhere from 5 minutes to 1 hour. This means that even if the attacker steals the token, they can only use it for a short period of time, before they have to use the refresh token to refresh it. The security advantage of refresh token rotation</a></h4> When the client requests a new access token, it also receives a new refresh token. If a malcious client uses the same refresh token to retrieve a new access token, the server will detect that the refresh token has been used twice, and will revoke all refresh and access tokens. This means that the attacker cannot use the refresh token outside the application, as it will be revoked as soon as the user opens the app. Read more about refresh token rotation here</a>. My approach to web app development 2023-04-11T00:00:00+00:00 This blog post outlines the way I build web applications. As the title suggests, this is my approach, and is what works for me. This is a heavily opinionated blog post, but I hope that at least some concepts suggested here will be helpful for others. Web apps vs. websites</a></h2> The first thing to clarify is that web apps should be built differently from websites. Web apps are focused on functionality, while web sites are focused on content. Web apps may have a small amount of content, and web sites may have a small amount of functionality, but the main focus is different. Additionally, web apps are designed to be used more than once, while for web sites they may or may not be used regularly. This distinction is important to consider in order to optimise the user experience. In reality, there really is a spectrum between these two extremes. We will be focusing on the web app extreme of this spectrum. To server-render or not to server-render?</a></h2> A lot of the development happening with frameworks right now is about server rendering. By server rendering I'm referring to any type of rendering that happens remotely (not on the client). This definition includes SSR (server-side rendering), SSG (static site generation), and ISR (incremental static regeneration). On the other hand, CSR (client-side rendering) is rendering that happens on the client, which is the 'default' for most frameworks (React, Vue, Angular). There are three combinations of server-rendering and client-rendering (excluding when there are none): Server-rendered (no client-rendering)</a></h3> This approach is reminiscent of frameworks such as PHP/Django. The server renders the HTML and sends it to the client. The client then receives the HTML and displays it. The client does not render anything. Application is entirely stored on the server, through cookies or sessions. Data will not update without a refresh. Mutations are performed through forms which submit POST requests to the server, and cause the page to refresh. Progressive Enhancement</a></h3> This is the approach taken by Next.js, Remix, etc. The server renders the HTML and sends it to the client. The client then receives the HTML and displays it. This is good for performance and SEO, as data is displayed without needing JS to load. The client then performs a process called 'hydration', which means that it takes the HTML and turns it into a fully interactive application. The application state is stored on the client. Data will update without a refresh (with proper data fetching libraries). Mutations can be performed without refreshing the page. Client-rendered (no server-rendering)</a></h3> The server sends the client a blank HTML page, which references some scripts which render the page. The application state is stored on the client. Data will update without a refresh (with proper data fetching libraries). Mutations can be performed without refreshing the page. Often, the app uses client side routing, which means that navigation between pages happen without a refresh. These apps are often large, as the entire app is downloaded to the client, rather than a single page. After looking at these three, Progressive Enhancement seems like the most obvious choice. In progressive enhancement, data is fetched on the server, and then hydrated on the client. This means that the user gets the best of both worlds: the performance and SEO benefits of server-rendering, and the interactivity of client-rendering. PWAs</a></h2> Progressive Web Apps (PWAs) are applications built using web technologies to deliver experiences that rival those of native apps. A key capability of PWAs is that they use service workers. Service workers are a type of web worker that runs in a separate thread, and among their capabilities is the ability to cache and intercept network requests. Web applications should behave like native apps. Native apps don't download their entire codebase from a server every single time they are opened. Native apps don't need to be connected to the internet to work. If web apps are to be viable alternatives to native apps, they need to have these capabilities. The web applications I build rely extensively on service workers. On first load, the web app is downloaded from the server and cached. On subsequent loads, the web app is loaded exclusively from the cache. When there is an update, the new code is downloaded in the background. When the user refreshes the page, the new code is loaded from the cache. For this to work, there needs to be some kind of client-side rendering, as the app is largely decoupled from the server. This automatically rules out the PHP-style approach, with no rendering on the client. Progressive Enhancement or CSR?</a></h2> The question now is whether to use progressive enhancement or client-side rendering. For the reasons outlined above, progessive enhancement will often provide the best user experience. However, for PWAs, progressive enhancement has diminishing returns. Consider that for devices supporting PWAs, the client only loads the app once, and then it is cached. This means that the benefits of progressive enhancement are only gained on first load. And since apps are used numerous times, these benefits are not as important, as for the vast majority of times, progressive enhancement has no benefit. caniuse.com</a> estimates that 96.4% of users globally support service workers. As nice as it would be to support the remaining 3.6%, for most developers, the cost of supporting these users is not worth it. Architectural Divergence</a></h3> As outlined in Remix's</a> article, architectural divergence leads to extra complexity, which leads to poor DX (developer experience). The architectural divergence Remix is talking about is about the divergence between SSG and CSR. Remix's solution is to fetch data on the server through SSR only. This is a great solution for websites, but as outlined above it doesn't work for PWAs. The architectural divergence in progressive enhancement is the initial fetching on the server, and the partial updates on the client. This leads to needing to write logic and data fetching code that runs both on the server and client, leading to an increase in complexity caused by architectural divergence. As a result, I opt to go for CSR-only for PWAs. This removes the additional complexity of data fetching on the server, which leads to a better DX. Progressive enhancement may be an ideal solution, but unless you have the time and resources to support the 3.6% of users who don't support service workers, it is not worth it. Making SPAs work well</a></h2> There are a lot of issues with SPAs. Here are the two most common issues: SEO - Search engines can't crawl SPAs.</li> Performance - SPAs are slow to load, and slow to run.</li> </ul> SEO</a></h3> Many people point to SEO as an issue for SPAs. However, we are talking about using SPAs for apps, not websites. Since apps are about functionality, not content, there is not much for the search engine to crawl in your app anyway. As for discoverability, you should build a website landing page to promote your app, using SSR or SSG. This landing page should be discoverable by search engines, and should link to your app. Performance</a></h3> This isssue has two sub-issues: initial load time, and runtime performance. Initial load time</a></h4> SPAs tend to be quite large, as they contain the entire app and all of its pages. This can easily lead to bundle sizes in excess of several megabytes. However, since we are building PWAs, this download only needs to happen once, as it is then cached. Additionally, through code-splitting, we can ensure that on first launch, only the code needed to render the current page is downloaded. This makes initial load time much better. And even if after this, load times are still bad, it only needs to happen once, so the impacts are minimal. After this initial load, service workers should start caching the rest of the app in the background. Runtime performance</a></h4> Since the app is using CSR, all of the rendering logic happens on the client. This moves the bottleneck from the network to the CPU. This is usually good, since for most devices it is faster to do things locally than to wait for the network. However, for extremely slow devices, this can be an issue. There is no easy answer to this problem, but generally performance issues are caused by poorly written code, and can be solved by refactoring. Ultimately, you should tailor your apps to work well for most of your users. You should collect performance metrics such as Core Web Vitals, and use them to make informed decisions about what to optimise. If most of your users are on devices that cannot support an SPA approach, than you should consider another approach. Data fetching</a></h2> One of the key principles I use when building apps is to have some data available as soon as possible, and try to keep that data up to date. Having data available as soon as possible, can be accomplished by having a cache, so that the data can loaded from the cache quickly. Keeping this data up to date could be accomplished by refreshing the data on a certain interval, and doing it whenever the user refocuses on a page. This can be quite hard to implement, so libraries such as SWR</a> and TanStack Query</a> exist to make this easier. Conclusion</a></h2> In this article, I outlined the pros and cons of the three main approaches to building web apps: SSR, CSR, and progressive enhancement. I then outlined the pros and cons of using progressive enhancement for PWAs. I concluded that for PWAs, CSR is the best approach, as it removes the complexity of data fetching on the server, and leads to a better DX. I also outlined some of the issues with SPAs, and how to solve them. Finally, I outlined how to keep data up to date. Keep in mind that this is just how I build apps, and you should do what leads to the best user experience for your users.