For the past four years, as a background side-project I’ve been (more slowly and sporadically recently) working on developing sitemon3, a re-design and re-write of sitemon2 as a distributed system (rather than the monolith the previous version was), whilst also writing it in Rust as another vehicle to help learn and get better at Rust - but also to try and (re-)learn some of the workings of modern HTTPS, web development and SQL/databases due to what knowledge I previously did have in those areas having atrophied quite considerably over the past 15 years by virtue of being in the VFX / computer graphics industry working on other problems that generally didn’t require that type of knowledge (although still with occasional basic usages of web service APIs and various databases).
The web has seemingly changed quite significantly in that time, with many more websites now using client-side rendering, in addition to many websites now being behind CDNs (like Akamai and Cloudflare) that often are configured with “bot detection” via methods like TLS- and HTTP-handshake fingerprinting, and because of this, it’s now not as easy to just use vanilla libCURL to request pages and emulate web browsers. Instead using a full browser engine really needs to be used. This is not the end of the world, however I still personally wanted to develop a libCURL-based ‘engine’ (in addition to a full browser one) for requests.
So I decided recently it might be interesting to play around with making a very basic (at least initially) website that was designed explicitly with the purpose of being able to test basic HTTP user agents. There are several applications that exist that allow doing this locally (mostly designed to test HTTP API endpoints) and run as servers, however I decided to make something basic from scratch mostly for my own education and experience that is
live on the internet.
So I’ve created http-test.com as an initial website which has some basic static responses currently. Presently it’s implemented using a Caddy config, but I also have it implemented using a self-written HTTP/1.1 webserver (with both C++ and Rust ports), which in the future I plan on using (likely still behind a reverse-proxy) to handle dynamic and configurable responses (i.e. like every 10 requests with a token, return an error to test things failing correctly - or random responses).
For close to two years now I’ve been developing my own webserver, partly to re-learn about HTTP and web technology that I haven’t really been following too closely for over eight years, and therefore my knowledge of that area has atrophied, but also to be able to have a stand-alone webserver that I control and can customise to my needs within one stand-alone codebase, rather than having to use multiple technology stacks, which from my point-of-view is a huge benefit (much less configuration and things to learn/remember). My main use-case for having a webserver is developing my own photo gallery with photos described in various ways (by date, geographical location hierarchy, tags, camera type, etc) which off-the-shelf/available solutions don’t provide to the degree I want.
I’ve now got a solution which supports HTTP 1.1 fairly robustly that I’m relatively happy with from a C++ technology perspective, although the JavaScript/HTML/CSS parts - of which I’m not very experienced - are pretty basic and likely less-than-ideal in terms of good practice and the best ways of doing things.
For hosting things on the public web, HTTPS support is generally a very good idea these days, and given there’s authentication / password functionality for my photo gallery, having HTTPS support is critical to prevent sniffing HTTP headers (among other security risks). Generally, it’s not the ideal situation to
put a self-created webserver on the public internet directly with public ports open: it would better practice to put it behind a well-known and reliable web/proxy server like nginx or Apache - however I was curious what would be involved in implementing HTTPS secure socket functionality at both the HTTPS / TLS and TCP levels, so decided to start implementing HTTPS support a few months ago.
At first I had intended to go with OpenSSL directly, however a bit of research seemed to indicate that might not be the best solution given the complexity and breadth of OpenSSL’s functionality, and therefore the possibility of making mistakes with utilising it would be greater, leading to more potential bugs.
There were a few options I could have taken, although some of the software licenses did not entirely match my ideal choices in that area, but in the end I decided to use AWS’s s2n. It’s a minimal open-source TLS implementation, although it does rely on other libraries (i.e. OpenSSL, but it can support others) for the cipher side of things - but it’s simple to use, and seems to have sensible defaults.
Implementation wasn’t too difficult, although generating certificates for localhost is annoying for testing, and not all web browsers were happy with them (Chrome refused to accept them, but Firefox did). It’s also a lot more troublesome to debug what’s going on at the TCP/HTTP level, as you can’t as trivially connect WireShark and sniff what’s going on at the packet level (but that’s the whole point!).
So my webserver (creatively called ‘WebServe’) now supports both HTTP and HTTPS connections, the latter with certificates authenticating the connection, theoretically allowing perfectly secure connections between the web browser and the webserver. It seems to work perfectly with Firefox and Chrome on Linux and MacOS, however very occasionally Safari on iOS (but not MacOS) fails to download an image when keep-alive is enabled, so it’s possible there’s some subtle bug somewhere still.