(Approx.) Great Britain Electricity Generation charts

Since I got back to the UK, I’ve been trying to understand and take more of an interest in what’s going on here again (although unfortunately, that’s often somewhat depressing across the political spectrum), and one area I’ve been looking into is Electricity generation (and prices!) for Great Britain.

There’s a large amount of renewable electricity generation from Solar and Wind these days, but it’s obviously highly variable with the weather, the time of year (i.e. day length for Solar) and cloud cover, so Natural Gas generation is still used to “top up” demand (often to very significant proportions), but these days Coal and Oil don’t appear to be used any more (Gas is the cleaner alternative and can be spooled up very quickly when needed).

There are a number of nice live dashboards people have made showing the data summary - National Grid: Live and Live GB Electricity Generation being two good ones, and there are free data feeds from a variety of different sources (although some of the values are estimates for certain things).

Very few of the existing dashboard sites appear to have detailed long-term historical chart data - they seem to either approximate it with daily averages further back in time, or just limit the history duration - and the fidelity of some of the charts is often quite limited due to filtering (somewhat understandably given some of the limitations of the source data), so I decided to try setting up a data storage and chart generation setup to scratch this itch for myself.

Stacked area chart:

Line plot chart:

Given I also wanted high-fidelity charts (non-interactive for the moment), I decided to use Python for this given how nice matplotlib charts are (especially when using seaborn styling), as I had also been meaning to dust some of the cobwebs off my Python skills that I hadn’t seriously used for almost two years, and I was similarly keen to play around with some of the newer Python tooling like uv, which is a more recent Python package and project management tool, which is written in Rust and is much faster than PIP.

There’s no single central source for a complete breakdown of electricity generation source data in Great Britain - the closest to that seems to be Elexon who provide regular snapshots (every 5 minutes) via data APIs of various different aspects of electricity generation and interconnect transfers, however there’s no single API that returns all generation sources, and some data sets are non-complete, in that they don’t include full Wind power numbers and don’t have any Solar power numbers. There are forecast APIs which estimate these numbers more accurately, but they’re only updated every 30 mins, so in the end I decided to use a different source for the Solar (PV) generation which is Sheffield Solar which is also an estimate, but seems to be updated more regularly.

For the moment, I’m using those two data sources for all generation types - note however this does mean that the numbers for Wind generation are lower (often by around 15%-20% I think) than the actual true Wind generation values, as the source data doesn’t included “embedded” wind farms. I’m also not including any of the Interconnector transfer amounts (which can be positive or negative depending on the direction of transfer) to neighbouring countries, which means the charts aren’t an accurate display of Demand of electricity generation in Great Britain.

So I now have an application setup where data is gathered every 5 minutes from the data source APIs, stored in an SQLite DB, and then I can query the data from a FastAPI-implemented API, which is then fed into matplotlib to plot the above charts locally. The server is running in a Podman container.

I think the stacked area chart looks nice, but obviously suffers from the typical read-ability issues stacked charts can suffer from in terms of gauging absolute magnitude, especially when offsets to lower data series happen significantly in a short space of time, and distort or squeeze the rest of the chart, but it does allow to roughly see overall trends during the day and through a week.

I am tempted to in the future try and additionally store some weather data (temperature and wind speed) for a variety of different locations around the country to see if plotting those against the Generation values (Demand values would arguably be better) would show likely correlations, i.e. more electricity use when it’s colder for heating (probably not that much AC in the UK) and more wind generation when it’s windier.

I’d also like to play around a bit with interactive JS/Canvas charts and see if it’s possible to have something that’s close to the presentation and fidelity of the matplotlib charts.

Process Recorder - psrec

I’ve been progressively trying to learn the Rust programming language for around a year and a half now and as well as porting some of my existing C/C++ apps I have to Rust as learning exercises (but with the C/C++ versions still mostly being the main ones for the moment) I have also started to write some new from-scratch apps in Rust when I think that makes sense, rather than defaulting to C/C++ as I previously would have.

I’d recently had the need to record some basic app process stats (CPU usage and RSS memory usage) over the duration of its running, and while there are existing applications out there - i.e. psrecord - that would largely do what I wanted, I was tempted to try and write my own version in a “native” language - at least for the recording part: the plotting / visualisation part is a bit more tricky. This was partly so as to have another small project with which to gain more experience with Rust, but also because I wanted additional features like the ability to control whether to have “normalised” or “absolute” CPU usage, and the ability to separate CPU time into user and system time - so I’ve written an initial version of psrec which is my own equivalent to psrecord written in Rust.

This initial version is pretty basic so far, and doesn’t yet support all the additional features I wanted: I’m making use of the psutil crate to extract the process information for the moment, but its functionality is incomplete as it doesn’t support extracting info on a process’ children, or info like the number of active threads a process has, which is functionality I will likely want at some point, so I’m no doubt going to have to get down in the weeds with the /proc/<pid>/ file system interface, which can sometimes seem a bit primitive and messy in my experience (it would be nice to have a first-class API to access the data efficiently, rather than doing string parsing, although it does make things very visual and easy to debug).

For the moment, I’m using Python and matplotlib to plot the resulting data, which to some extent is a bit at odds with writing the main app in a compiled language like Rust, but I think it’s okay for the moment, and it allows the recording app to theoretically be more efficient and low-overhead (although polling the /proc/ file system and recording samples every second isn’t really that much overhead), whilst using Python for things it’s very good at.

Below is a basic example of the chart plot of a quick process recording: