(Approx.) Great Britain Electricity Generation charts

Since I got back to the UK, I’ve been trying to understand and take more of an interest in what’s going on here again (although unfortunately, that’s often somewhat depressing across the political spectrum), and one area I’ve been looking into is Electricity generation (and prices!) for Great Britain.

There’s a large amount of renewable electricity generation from Solar and Wind these days, but it’s obviously highly variable with the weather, the time of year (i.e. day length for Solar) and cloud cover, so Natural Gas generation is still used to “top up” demand (often to very significant proportions), but these days Coal and Oil don’t appear to be used any more (Gas is the cleaner alternative and can be spooled up very quickly when needed).

There are a number of nice live dashboards people have made showing the data summary - National Grid: Live and Live GB Electricity Generation being two good ones, and there are free data feeds from a variety of different sources (although some of the values are estimates for certain things).

Very few of the existing dashboard sites appear to have detailed long-term historical chart data - they seem to either approximate it with daily averages further back in time, or just limit the history duration - and the fidelity of some of the charts is often quite limited due to filtering (somewhat understandably given some of the limitations of the source data), so I decided to try setting up a data storage and chart generation setup to scratch this itch for myself.

Stacked area chart:

Line plot chart:

Given I also wanted high-fidelity charts (non-interactive for the moment), I decided to use Python for this given how nice matplotlib charts are (especially when using seaborn styling), as I had also been meaning to dust some of the cobwebs off my Python skills that I hadn’t seriously used for almost two years, and I was similarly keen to play around with some of the newer Python tooling like uv, which is a more recent Python package and project management tool, which is written in Rust and is much faster than PIP.

There’s no single central source for a complete breakdown of electricity generation source data in Great Britain - the closest to that seems to be Elexon who provide regular snapshots (every 5 minutes) via data APIs of various different aspects of electricity generation and interconnect transfers, however there’s no single API that returns all generation sources, and some data sets are non-complete, in that they don’t include full Wind power numbers and don’t have any Solar power numbers. There are forecast APIs which estimate these numbers more accurately, but they’re only updated every 30 mins, so in the end I decided to use a different source for the Solar (PV) generation which is Sheffield Solar which is also an estimate, but seems to be updated more regularly.

For the moment, I’m using those two data sources for all generation types - note however this does mean that the numbers for Wind generation are lower (often by around 15%-20% I think) than the actual true Wind generation values, as the source data doesn’t included “embedded” wind farms. I’m also not including any of the Interconnector transfer amounts (which can be positive or negative depending on the direction of transfer) to neighbouring countries, which means the charts aren’t an accurate display of Demand of electricity generation in Great Britain.

So I now have an application setup where data is gathered every 5 minutes from the data source APIs, stored in an SQLite DB, and then I can query the data from a FastAPI-implemented API, which is then fed into matplotlib to plot the above charts locally. The server is running in a Podman container.

I think the stacked area chart looks nice, but obviously suffers from the typical read-ability issues stacked charts can suffer from in terms of gauging absolute magnitude, especially when offsets to lower data series happen significantly in a short space of time, and distort or squeeze the rest of the chart, but it does allow to roughly see overall trends during the day and through a week.

I am tempted to in the future try and additionally store some weather data (temperature and wind speed) for a variety of different locations around the country to see if plotting those against the Generation values (Demand values would arguably be better) would show likely correlations, i.e. more electricity use when it’s colder for heating (probably not that much AC in the UK) and more wind generation when it’s windier.

I’d also like to play around a bit with interactive JS/Canvas charts and see if it’s possible to have something that’s close to the presentation and fidelity of the matplotlib charts.

Process Recorder - psrec

I’ve been progressively trying to learn the Rust programming language for around a year and a half now and as well as porting some of my existing C/C++ apps I have to Rust as learning exercises (but with the C/C++ versions still mostly being the main ones for the moment) I have also started to write some new from-scratch apps in Rust when I think that makes sense, rather than defaulting to C/C++ as I previously would have.

I’d recently had the need to record some basic app process stats (CPU usage and RSS memory usage) over the duration of its running, and while there are existing applications out there - i.e. psrecord - that would largely do what I wanted, I was tempted to try and write my own version in a “native” language - at least for the recording part: the plotting / visualisation part is a bit more tricky. This was partly so as to have another small project with which to gain more experience with Rust, but also because I wanted additional features like the ability to control whether to have “normalised” or “absolute” CPU usage, and the ability to separate CPU time into user and system time - so I’ve written an initial version of psrec which is my own equivalent to psrecord written in Rust.

This initial version is pretty basic so far, and doesn’t yet support all the additional features I wanted: I’m making use of the psutil crate to extract the process information for the moment, but its functionality is incomplete as it doesn’t support extracting info on a process’ children, or info like the number of active threads a process has, which is functionality I will likely want at some point, so I’m no doubt going to have to get down in the weeds with the /proc/<pid>/ file system interface, which can sometimes seem a bit primitive and messy in my experience (it would be nice to have a first-class API to access the data efficiently, rather than doing string parsing, although it does make things very visual and easy to debug).

For the moment, I’m using Python and matplotlib to plot the resulting data, which to some extent is a bit at odds with writing the main app in a compiled language like Rust, but I think it’s okay for the moment, and it allows the recording app to theoretically be more efficient and low-overhead (although polling the /proc/ file system and recording samples every second isn’t really that much overhead), whilst using Python for things it’s very good at.

Below is a basic example of the chart plot of a quick process recording:

Interpreter Programming Language Benchmarks

I’ve sporadically but progressively been continuing to work on my Mint interpreter programming language/VM that was heavily based off Robert Nystrom’s excellent Crafting Interpreters Lox language tutorial, and have spent a fair bit of time trying to optimise the main VM bytecode despatch loop among other areas of the VM, as well as performing some benchmark comparisons with other similar bytecode VM interpreter languages like Lua, Python and Ruby to see where Mint stands performance-wise as well as to understand how fast the other VMs are, how they work and some of their performance characteristics for different things.

I managed to speed up the original HashMap implementation a bit by caching the capacity mod bit-mask and the capacity and table load ratio, so as to not recalculate them on-the-fly every lookup which wasn’t needed, and to only update them when the table resized, in addition to adding some compiler hints in order to force inline things and give hints on branch prediction likelihood which helped a bit more.

One of the main bottlenecks in raw execution speed is the main VM stack in terms of operations pushing and popping items onto and off it, so this is really the hot loop of the VM - at least when the Garbage Collector is not being exercised. On the bytecode opcode execution side of things, I added some new opcodes in order to do some things more efficiently, for example to operate directly on items within the stack in certain limited scenarios when constant values are used, without having to pop and push them back - for example this can be used by compound assignment operators like +=, -=, *=, etc, with constant values being used as the right-hand-side argument, and helped speed things up a bit.

For general for loops incrementing a value by 1 each time, the push/pop overhead was fairly substantial, and I added a dedicated OP_INC_LOCAL opcode to help with this, which can increment the top of the stack by one without popping the stack and then having to push the result after adding 1. The compiler part can detect when this can be used (e.g. in the for loop increment clause, with a constant value being used as the increment) and will use it when possible.

Lua - being a register-based interpreter VM - seems to have an advantage in the area of performance - especially in maths-heavy code - because it doesn’t need to do as many stack operations, and this is one of the reasons Lua is one of the faster bytecode interpreter languages.

The main dispatch loop can also be optimised by using “computed gotos” instead of just a case statement for each opcode within a switch statement: with switch / case statements, compilers are obliged (in C and C++ anyway) to add an upper bound value check, which slows down execution.

In terms of where things stand now performance-wise, I’m fairly happy: Mint is not the fastest interpreter language VM, but it is very competitive against Python 2, Python 3 and Ruby in most cases, and occasionally can win against Lua.

For performance comparisons, I’ve mostly been using basic problem solving exercises from Project Euler which are normally maths-heavy, meaning the benchmark test programs I’m using are slightly skewed benchmarks towards maths operations compared to more general operations (i.e. they don’t generally exercise the Garbage Collector infrastructure), but that’s the type of code execution I’m mostly interested in for the moment.

I’m going to compare Mint against four (I’ll compare Python 2 and Python 3 separately) other interpreter language VMs:

• Lua 5.3
• Python 2.7
• Python 3.8
• Ruby 2.7

These are the package versions available with the Linux distro I’m using (Linux Mint) on my laptop, and I’m going to compare them against Mint (VM) built with GCC 9.3, as that’s the system compiler version that I assume the packages for the other interpreter language VMs were compiled with, so seems the fairest compiler to use to compile with. Mint will be built with optimisation level -O3. Normally I like also comparing different compiler versions and optimisation levels, but I’m not going to do that this time as my aim is to compare against other interpreter VMs, and that would increase the number of tests I’d have to do quite a lot, and I’d rather not get into building each one from source, but that would be the fairest way of doing a fully-comprehensive comparison if I had the time.

The tests are being done on a laptop with an Intel i5-8350U processor, with the power plugged in. I’ll make sure the CPU temp is normal before running each test to prevent thermal throttling, and will run five identical tests of each benchmark with each interpreter language, using the mean average as the final result.

I will run the following seven tests:

As quick partial examples, here is the Mint script code for Test 3:

var a = 1;
for (var i = 0; i < 100000000; i += 1)
{
    a = (i + i + 3 * 2 + i + 1 - 0.42) / a;
}
print a;

here is the Lua version of Test 3:

local a = 1
for i = 0,100000000-1
do
    a = (i + i + 3 * 2 + i + 1 - 0.42) / a;
end
print (a)

here is the Python 3 version:

a = 1
for i in range(0, 100000000):
    a = (i + i + 3 * 2 + i + 1 - 0.42) / a
print(a)

and here is the Ruby version:

a = 1
0.upto 100000000-1 do |i|
    a = (i + i + 3 * 2 + i + 1 - 0.42) / a
end
print a
print "\n"

These are the results, showing average execution time for each test for all interpreter language VMs (smaller values are better):

Lua has a very good showing (largely I think because it’s a register-based VM instead of a stack-based one), and I’d expect the non-JIT version of LuaJIT which is heavily-optimised on x86-x64 would be even faster.

Mint just wins in Tests 6 and 7, and is second fastest in the remaining tests, which isn’t bad as far as I’m concerned. There’s more room for optimisation - especially with regards to the Garbage Collector which these benchmark tests purposefully didn’t exercise, but also in other areas, i.e. Constant Folding is on my list to look at implementing - but I’m happy enough with the improvements and speed for the moment.