What this is

I made a custom circuit board that kind of zombifies the original standing desk controller. It is able to read the current height of the desk from the original controller, as well as pilot the desk up and down. It is controllable via HTTP requests, as well as from buttons on the front of the controller.

GET http://esp32-abcde/desk
{"height":720}

GET http://esp32-abcde/desk?height=780
`OK`

Initial exploration

The standing desk I have is a VIVO DESK-V122EB frame with a 2.5m IKEA countertop placed atop it. More importantly, the controller box my standing desk has looks like this (Apologies for the terrible photo, I didn’t take any photos before doing what I did, so this one was cropped from a much larger photo).

After doing some Googling, nothing seemed particularly relevant to my specific standing desk, and other solutions I found involved strapping motorized button pressing robots to the controller, so I decided to open up the controller box and take a look to see if I could do something smarter.

Looking at the circuit board, I realised it would be trivial to ‘press’ the up and down buttons from an external microcontroller. All I’d need to do is use a transistor to bridge the contacts of the switch. I really wanted something a bit smarter than that though. I wanted whatever I built to be able to know the height of the desk. After studying the board for a bit and googling the random part numbers on the chips I could see on the board, I identified the AIP650 as the chip responsible for driving the LCD. Fortunately, a data sheet was available for this part, but unfortunately it was in Chinese, a language I do not speak.

According to the data sheet, translated with DeepL, it receives data in an i2c ‘like’ format, and then drives a 3 segment LCD just like the one on this board based on what data it received. I was unable to determine from the data sheet what the data being sent to the controller actually “meant”, but wasn’t worried about that for the moment. I just wanted to capture the data.

For those not familiar with hardware, I2C is the scheme by which most circuits use for inter-chip communication. In my desk controller, there is a microcontroller which writes data to a LCD driver chip, which in turn lights the correct segments on a 3 digit segmented LCD. How would I capture the data though? Easy, I thought! All I had do to was hook up to the same two pins the data sheet reported as being the i2c pins along with the grounds of the standing desk controller and my esp32, and I could then “sniff” the data that was being sent across the i2c bus, decode it, and then use that for something useful.

So away I went! I soldered two wires directly to the pins on the chip that were marked as the data pins (I am deeply sorry to anybody offended by my soldering job!), and soldered a third wire to a ground point on the board. I then connected those wires to an ESP32, and then went looking for somebody elses code to try to sniff the i2c data to see if this was even possible.

I came across one such project online, which just about worked. It would output what looked like plausible i2c data, but would constantly crash or start outputting total nonsense. At this point, I hadn’t really understood I2C, nor did I understand the code here at all, so the fact it worked even a little bit I took as a win.

At this point, I still hadn’t really considered how the code worked at all, so I then took the plunge into the code and figured out its method of operation, so next I took a dive into the world of I2C.

A quick look into I2C

If you understand I2C even a little bit, you probably understand it better than I, so you can skip this section.

In I2C communication, there are two pins involved. One is the clock pin, usually referred to SCL, and the data pin, referred to as SDA. The participants on an I2C bus are known as controller devices, and target devices (although are commonly referred to as master/slave in data sheets and the such!). Controller devices write to the I2C bus, and target devices listen for writes that are targeting them. Target devices do respond on the I2C bus, but this is to just acknlowledge they correctly received data. Remember that this is digital data - so the controller can only pull either of the pins high or low, which literally means “emitting voltage” or “connected to ground”. It emits a clock signal (continuous pulses on the SCL pin), along with varying pulses on the SDA pin for signalling.

Let’s review one I2C communication session:

1. A controller wants to communicate with a target. IT first must assert control of the bus. To do this, it first pulls SCL high, and then pulls the SDA line high, then pulls it low. This transition of SDA from high to low while SCL has been held high implies a master is about to broadcast data, and targets know to listen for incoming data.

2. The controller needs to let all those target devices listening know which device it wants to talk to. It does this by sending 7 bits of data which correspond to a device address, along with 1 bit to indicate whether it would like to read or write to that specific device. It transmits these bits of data by pulling SDA either high or low to indicate a 1 or 0, along with a clock pulse (pulling the clock from low to high). This triggers the target device to read the value that is present on the SDA pin, thus transmitting a byte.

A note about addresses: They’re not always 7 bits, and sometimes can be 10 bit. They’re usually hardcoded into whatever target device is receiving them - and for hobbyist grade Arduino attachments, you can even configure the address through jumper pins or switches, allowing you to connect more than one of the same device to an i2c bus.

3. Next, the target with the correct address must ACK this transmission, to signal to the controller that it is ready to receive data. The target does this by pulling the SDA pin low, which the controller interprets as an ACK. If no target ever does this, the controller should assume it wasn’t heard for some reason.

4. Assuming the target ACKed the transmission, meaning it recognised the next data on the I2C bus is intended for it, the controller proceeds to transmit a data packet. IT transmits 8 bytes of data following the same scheme as the address transmission above, and again waits for an ACK condition the same way. If there is more than a single byte of data that needs to be transferred, it just transmits the next 8 bits after receiving the ACK. This cycle can happen an arbitrary number of times.

5. The controller is now done transmitting to that particular I2C device, so it needs to signal that. It sends a stop condition. This is similar to the start condition described above, except this time we pull SCL high, and then pull SDA high. The transition of SDA from low to high while SCL was held high indicates the transmission is over.

Digging deeper into the desk

Knowing all of this now, I decided to refactor the code sample above into a form that was better understood by me, which ended up turning into this project.

A quick discussion on how this code works. Since we only really care when the state changes on the pins we are sniffing (SDA and SCL) we can get away with using interrupts. An interrupt causes the CPU to halt execution of whatever it was doing, switch to some other context, do some other processing, and then return back to the main program loop. In our context, we want interrupts that trigger on state changes for the I2C Pins. Realistically, there are only two events we only really care about:

How we detect start and stop conditions: SDA changing from HIGH > LOW or from LOW > HIGH triggers the interrupt i2cTriggerOnChangeSDA(). If SCL was LOW, this isn’t a start or stop, so just return. If SCL was high, depending on the current bus state (discussed later), along with whether SDA is high or low, we record a Start or Stop to a buffer. Concerning the bus state: we initially assume the bus is IDLE, and move to TX if there is a start condition detected. If a stop condition is detected, we move to IDLE)

How we detect data being transmitted: SCL changing from Low > High indicates that there is data for us to read. so, the transition triggers the interrupt i2cTriggerOnRaisingSCL(), which checks the bus state. If it was set to TX by i2cTriggerOnChangeSDA(), we read whatever data is present on SDA pin and write it into the buffer

This results in a stream of data, and very quickly a buffer overflow. To resolve this, we need some code that processes that buffer and does something useful with it. We can only do this processing with the I2C bus is idle, so, in the main program loop, we wait for the IDLE bus condition to be reported by the interrupts above. Once we have that, we can safely parse through the buffer, and do something with the data. In the example code above, it just prints whatever it got to Serial. Once it’s caught up with the last written position of the buffer, it resets the variables that keep track of where the buffer was last written to and read to, so that we can continue to use that same memory over and over.

The code did as I asked and outputted streams of 1s and 0s along with start and stop conditions to the screen, but I had no idea what any of it meant, nor if the code even worked. What I could see is that there seemed to be bursts of data, and if the data was to be believed, at least 5 I2C addresses receiving data.

I lamented this fact in a conversation with a colleague (if you’re reading this, you know who you are, THANKS!), and he suggested I pick up a Logic Analyzer. I had no idea what a logic analyzer was, but for years I’d looked on in envy as others used their logic analyzers for interesting hacking related tasks, and thought that stuff was way too complicated for me. I’d never gotten one because I was afraid of buying it, realizing I was way out of my depth, and it becoming “that thing I bought that I have no idea how to use”.

This time however I bit the bullet and bought the cheapest logic analyzer I could. I had no idea how to use the thing when it arrived, but I discovered that Pulseview is a piece of software that can be used for logic analysis. I connected the SDA and SCL pins of my desk up to my logic analyzer, connected it to my computer, fired up Pulseview, and clicked around a bit. Almost immediately, I figured out how to “sample” the data the logic analyzer was getting, as well as that Pulseview includes a super handy I2C decoder built right in!

So, I sampled, and lo and behold, roughly what I’d been observing with the esp32 sniffing code was visible here too! Short, sporadic bursts of data. Again, sorry for the photos, you might want to right click and open them in a new tab so you can arbitrarily zoom them!

Zooming in, even more success! Comparing what my code was outputting to what was being seen on the logic analyzer, I could see the same addresses and the same data packets. Incredible. It seemed that in my particular case, each address received exactly one byte of data, and that the NACK/ACK bits didn’t seem to make a lot of sense. I wonder if this is what the creators of the AIP650 meant when they said it was ‘i2c-like’.

I still had no idea what any of this meant, so I flailed about for a bit. After a bit of thinking, I theorised that perhaps the AIP650 chip was not “one” i2c device, but instead was at least three, one per segment. That way, the microcontroller that was asking it to display stuff could update each segment as it needs to. It kind of made sense. A segment of the LCD consists of seven segments, along with a period that appears beside that segment. That’s 8 bits of information, so maybe each segment is linked to a specific bit in the byte.

To test that theory, I got the rightmost segment to display a zero, and sampled using the logic analyzer. I dumped the data out of Pulseview, and proceeded to make the segment display a 1, rinse and repeat all 9 digits. I wrote all the data I got into a spreadsheet (this took a while) in the hopes I could spot a pattern.

This allowed me to confirm that a specific pattern of bits corresponded to the numbers 0 through 9, and also allowed me to identify which i2c address corresponded to which position on the display (the digit), as well as allowing me to identify that the most significant bit in the byte corresponded to the period portion of the digit. This is decent progress, I thought!

What it didn’t do however was tell me which bit corresponded to which individual segment of a digit. Figuring that out was kind of like solving a Sudoku. I sat there with a sheet of paper, drawing the digits as segments 0 through 9, trying to find the commonalities.

As an example of what this particular puzzle took, let’s work through identifying two segments of a digit. We see that when the digit 1 is present on the LCD, the binary 00000110 is present. That must mean that the 6th and 7th bit correspond to the segments B and C, but we don’t know which is which. To determine which, we now have to find a digit which only had either B or C illuminated.

The digit 5 proves to be a good candidate here, since C is illuminated, but B is not. Looking at the data we collected, 5 seems to be represented by 01101101. The 6th bit is 1, the seventh is 0, meaning that the 6th bit must then be C, and B must therefore be represented by the 7th bit. Continue this process of deduction all the way and you’ll eventually figure out which bit represents which segment. In my case, each bit in the array represented the following segment:

• Position 0: DP
• Position 1: G
• Position 2: F
• Position 3: E
• Position 4: A
• Position 5: C
• Position 6: B
• Position 7: D

Now that I knew what the binary data meant, I was finally ready to decode it. I wrote some code that would parse each I2C frame in the buffer into the address portion to figure out which segment was being written to, and then parse the data portion into a character. I don’t have the code I wrote back then, but here’s the most recent revision showing how we turn the data portion into a digit, along with the code that parses through the buffer

Next up came actually raising and lowering the desk. I’d already done this sort of thing before, having previously tasked an ESP32 with pressing the button on my USB switching hub so this part was going to be relatively simple.

All you need to do really is take a transistor, connect the collector and emitter to your switch contacts, along with connecting the base to one of your ESP32s GPIO pins (through a resistor to limit current, 10k will do!). Then, in your ESP32 or Arduino code, all you need to do is configure your pin as an output pin, ie: pinMode(yourPinInt, OUTPUT);, and then when you want to press the button, you just bring the pin high by using digitalWrite(yourPinInt, HIGH);. To stop pressing the button, you bring the pin low digitalWrite(yourPinInt, LOW);.

The electronics assembly

On my particular standing desk, the switch contacts consisted of an outer and inner ring. I determined that the outer ring was positive 5v, and was common across the switches. The inner ring however was not common. Therefore, if I wanted my desk to go up or down, I needed to bridge ANY of the outer rings to the center contact of the up and down switch. I firstly tested my design on a Breadboard:

After validating this worked, I soldered together a protoboard. I firstly attached my ESP32. I then soldered two resistors to D33 and D32, and then connected the center pin of both transistors (the base) to each resistor. I connected the collectors of both transistors together (the leftmost leg, with the flat side of the transistor facing you), and connected those legs to the outer ring of one of the switches. I then connected each Emitter leg to the center ring of the up and down rings.

I also soldered D12 and D13 to the SDA and SCL pins on the AIP650 present on the desk controller, along with a wire to VIN which would later supply 5 volts, along with a ground pin.

I also attached wires to D15, D2, and D4, along with a second ground connection

• which I will later attach to three switches, allowing me to have buttons on the outside of my new controller to control the desk manually, as I could before. I didn’t bother connecting any of the other buttons of my desk to the ESP32 since I didn’t care about the functionality they exposed (presets and the such).

Once all this was complete, and it continued to work, it was time to write some code.

The code

My first revision of the code was a rats nest. I have no idea what I am doing in C++ (I’m a Golang/Ruby baby!), but its method of operation was very simple. It had two main variables responsible for control, the currentHeight, and requestedHeight. If they were not equal, the code would press the button that would cause currentHeight to move in the direction of requestedHeight, and release it once they were equal. Can you see the problem with this technique?

If not, no worries, I clearly didn’t see it coming either, and I really should have, given it’s a very common problem in DevOps land. We regularly deal with a concept called hysteresis, which is effectively a lag between an input and a desired output. If you’re more familiar with servers, maybe this example will help. Imagine you scale the number of replicas in a particular deployment based on a really simple metric like CPU utilisation. You might have some sort of rule like “If the CPU utilisation average hits 80%, add a replica. If it is below 80%, remove a replica. Then, when your clever autoscaler recognises that the average utilisation has exceeded 80%, it dutifully adds a replica. The utilisation then drops below 80%, and a replica gets removed, bringing the utilisation back up over 80%. This cycle repeats ad nauseum until you just use an autoscaler written by someone else.

Well friends, I had hysteresis in the form of a 2.5m long oscillating vibrating desk. The code frantically tried its best to make currentHeight equal requestedHeight, but all it did was cause my desk to oscillate up and down frantically until it eventually overheated and refused to do anything for around an hour after this event.

As hysterically funny as I found this, I really didn’t want to break my desk, so I went back to the code and tried to fix this.

The fix I came up with was stupid, but it works. The ‘proper’ solution is probably a PID controller, but the solution I came up with is far simpler to write and works perfectly, so I don’t really care. I just implemented a form of crude PWM on the button pushes! Conceptually, instead of holding the button until we see the target height, we instead hold the button until we are near the target height, at which point we start rapidly pressing and depressing the button, which has the effect of causing the desk to move more slowly. We keep doing this until we reach the target height, and then we’re done, without overshooting or undershooting the target.

Something else I learned the hard way is that my desk controller really does not appreciate having its button pressed and depressed at the clock speed of the ESP32, and it gets really confused and sad. This was simple to resolve though just by modifying the code to press and depress the button with a 50ms cycle time.

Through all this, I ended up with some code where I could program the ESP32 with a given height, and after I reflashed the ESP32, it would command the desk to a particular height.

I obviously wanted a way of setting this value arbitrarily, so I set up a simple HTTP web server that would do this for me, hosted on the ESP32. It sits there, waiting for requests, and if it receives a height change command, it does exactly as I described above and it worked beautifully.

3D printing an enclosure

CAD modelling is something I also have no professional experience with. I’ve been teaching myself over the last few months, and decided this was the perfect opportunity to model something relatively complex. What I wanted was a place to put the original controller board so the LCD was still visible, spots to put 3 real mechanical keyboard switches (yes, the kind you find in a keyboard :P), and a spot to screw in the protoboard construction documented above.

I came up with this. Beauty is in the eye of the beholder, I guess! I threw the job to my trust Sovol SV07 and waited a few hours. It turns out I designed something that is a nightmare to print, but the printer did an admirable job even with my ridiculous design. It’s impressive what modern 3D printers are capable of!

I then assembled everything pretty much as described, just with the addition of 3 mechanical keyboard switches, and it all went together mostly as designed. The print quality is frankly terrible. This was down to a misconfiguration of the printer, as well as the fact that this is a fairly difficult to print model. I was also being impatient and printing it at ludicrous speeds. 3D printing is a hobby that really benefits from patience, something I sometimes lack.

I have not released the STL for this particular enclosure, because it is an absolute nightmare to print, and has some design errors I haven’t addressed. If you really want it for some reason though, I will give it to you. Just ping me on Mastodon.

I’m finished, right?

Great! Install and done right? I thought so too. I installed it, and revelled in telling anyone who would listen that now desk go up and down when I send HTTP request to it! Cool right? RIGHT?!. It worked great.

Until disaster struck. A few hours after I installed it, my desk started going up on its own. I was happily coding away, and it started going up totally on its own. No command. WTF?!

I frantically unplugged it and started trying to think about why this happened. I racked my brain, but I couldn’t think of any reason. Through sheer luck it happened a second time while I was watching the serial output from the ESP32. I had it logging out the state of the currentHeight and requestedHeight variables if they changed, and the currentHeight randomly changed from 720 to 1520! I had focussed all my thinking about reasons why requestedHeight might change. I hadn’t considered the actual height of the desk was being misread.

It turns out, writing your standing desk controller the same way you write reconciliation loops in Kubernetes-land is not a great approach. It also turns out, that just by installing your project in an enclosure, you can introduce a brand new problem that was not present previously.

Enough foreshadowing, the problem was pretty obvious in hindsight. Sniffing I2C is not a perfect art. Electromagnetic interference can cause you to incorrectly read a 0 as a 1, or a 1 as a 0. In my case, if this happens, in the best case, we read an invalid I2C address, which just means we ignore that frame of data since we only care about a subset of i2c addresses (the ones that correspond to each portion of the LCD). In the next best case, the actual data portion gets corrupted somehow, and we get an arrangement of segments that does not correspond to a valid digit. In this case, we again just ignore that particular frame, and nothing bad happens. The most insidious cases though are where a data frame gets corrupted, and we end up reading the wrong digit.

Given my code works like a reconciliation loop, it notices that the current height does not equal the previously requested height, and frantically tries to correct for that; but of course, it’s reading invalid data, so desk go up even though I didn’t ask it to. Oh noooo.

I have made this situation extremely unlikely by adding a number of safety checks:

• If a new currentHeight comes in that is substantially different to the previous reading, we assume that new reading was invalid. Usually, the next frame of data is read correctly, so this catches most cases.
• If a new currentHeight comes in that is out of bounds (ie, is a value the desk should never display), we just ignore it. This happens quite frequently, since one of the most common errors I’ve seen is reading the first digit as a ‘1’ instead of a ‘7’ (a single bit different!).
• We only move if we have received a command to do so. CurrentHeight can change however it wants outside of a move command, we don’t care, unless the user has issued a move request. Once the desk has reached its requested height, we make sure the desk has stopped moving, and then stop caring about currentHeight changes until the next request comes in.

This has resolved that specific problem! It’s always good to remember to anticipate your hardware misbehaving and adding basic sanity checks to your code.

The code today

I reiterate from earlier, I do not really know C++/C or hardware, so this code is probably a litany of sins. Correct them in PRs if you like, I might even flash them to my desk, because those are words I can say without sounding insane now! So, when reviewing this code, bear that in mind. https://github.com/kn100/desksniffer/

But why?

Read this.

I’ve been working on a hardware project that involved connecting my standing desk to the internet. If I describe that project in pure utilitarian terms, its utility can be boiled down to “I want my desk to go up when I make a HTTP request to a specific endpoint”. How useful is this, really? Isn’t the previous solution of just leaning over to the right around 6 inches and pressing the button for a few seconds convenient enough for me? This is a perfectly reasonable question - if you assume my goal was to solve a real problem that mattered.

In the last few years, I’ve found that my love of tech has been dying a bit. Earlier in my life, I’d constantly experiment with Linux distributions, window managers, programming languages, text editors, hardware - really whatever was interesting that day. Spending the day compiling my kernel to test some silly tweak was never an issue. In the last couple years however I’d noticed that this experimentalist nature in me had died. I used a completely conventional Linux distro. I didn’t even bother to change the wallpaper. I had a bone stock phone. Electronics experimentalism was limited to projects that I could justify as having some real utility. If they weren’t useful, why would I waste my time on them? I learned Golang - and fell in love with its very utilitarian “get shit done” nature. No tabs versus spaces wars, no nine ways of doing the same thing, all of which are somehow not best practice according to somebody somewhere. I grew to dislike working in almost any other language.

More recently, that experimentalism seems to have bloomed in me again. I think that what originally killed it was a marriage to utility - the idea that everything I build should have some inherent useful output. Maybe I envision turning it into a startup one day. Maybe it solves a problem that others care about? Why experiment with a different Linux distribution when I can get by just fine with the one I have? Isn’t that just a total waste of time? Why bother solving some tiny niggle with a standing desk through weeks of research and learning - when the time could be better spent watching Star Trek? Maybe I could have just bought a different standing desk.

In addition, my love of tech is back in full force again. I’m now trying NixOS, a Linux distribution that bills itself a reproducible, declarative and reliable operating system. Some expressed ridicule when I mentioned this to them. Isn’t that just a waste of time? To some degree, they’re right. Everything I do on my new shiny NixOS install is possible on any other distro, and is almost certainly possible on a Mac. What they probably didn’t understand though is that this wasn’t a utilitarian decision. I didn’t switch to NixOS with any hope of improved productivity. I switched because I wanted to see what it was all about. I wanted to struggle to figure it out. I wanted to see what the benefits are. In that process, I have learned far more about how Linux works at a low level than I have in my years just sticking with the known working thing. The skills I acquired along the way will be helpful even if I decide to abandon NixOS one day. I didn’t switch because it was the prudent thing to do - I switched for switchings sake.

I recently acquired a 3D Printer. Many people buy a 3D printer with the goal of eventually turning it into a side gig - producing parts or whatever. I didn’t. I bought a 3D printer because I wanted to learn to model in 3D. I didn’t have a specific utilitarian goal in mind, but I felt that owning a 3D printer would push me to learn to design things in CAD in a way that doing it without having a way to make it physical would not. It did not improve my productivity in any way. My day job in DevOps is about as far from CAD as one can get - and I’m not considering any sort of career change. I don’t think I’ll ever design something that I intend to sell. I wanted to learn CAD because I thought it’d be fun, so I am, and I was right. It is fun, and it turns out 3D printers are hilariously useful machines to have around if you are willing to spend the time to use them properly. I didn’t buy a 3D printer because it solved a particular problem, or because I intend to become a product designer or artist. I bought it because I wondered how it worked.

So, revisiting the standing desk project one more time. Why did I do it? One skill I’ve always wished I had was knowing how to “sniff” data from consumer grade hardware so I could ship it off somewhere more useful. I’ve always wanted to develop some competency in C++ too - given it’s a language I’ve never had the opportunity to really write any significant amount of it. I looked around my desk, and decided that the nearest thing to me to had some data I could sniff was actually my desk. What if I could make it go up and down from HTTP requests? What if I could query its height from the internet? Thus, a project with very little apparent utility turns into an amazing puzzle, with endless learning opportunities. It involved learning some C++, how to operate a Logic Analyzer, how to decode i2c, some basic electronics, there was even a fairly hilarious reminder about hysteresis I’ll cover in a future post.

This might be obvious to some. I believe the reason why experimentalism like this captures my soul so completely when it does is because the goal is rarely the point. It’s the exploration. The learning. The sheer joy at making something work. The surprise when some previously insurmountable obstacle is surmounted. The dopamine rush as a whole new world of possibilities opens up to you. The new skills acquired along the way. The friends made when seeking help in random internet forums. It turns out that sometimes, the journey taken is far more important than the end goal.

I really hope I can keep feeling okay to taking my time working on things that have no obvious point, because the real answer to the question of why, is why not?

Other writing

• https://sigbovik.org/ the home of the Association for Computational Heresy - I specifically recommend YT: Harder drives as a good starting point. Imagine the worst storage mediums you can think of, and then marvel at the fact that these are somehow worse.
• http://antirez.com/news/123 - Antirez on the value of projects with “Hack value”.
• Hackers: Heroes of the Computer Revolution - Steven Levy - A fantastic book looking back at early hacker culture

The biggest most exciting use case for me with my new home server was a virtualised gaming environment. I’ve always wanted a living room gaming PC, but didn’t want to have a PC just for that purpose. I’ve had a home server of some sort for the last couple years, usually running some Docker containers, and I always wondered if one day, I could run Windows in a VM, with a graphics card. I can confirm that, yes, it is possible, and it works absolutely great. Before I discuss how to set this up, Let’s discuss what it’s like.

What’s it like?

In short, it’s great! In my setup, I’ve put the server in a cupboard, and have run a rather long fiber optic HDMI cable from the cupboard out to a television I have in my living room. I then control it using a small media centre style keyboard and mouse, as well as an old Xbox 360 controller to use when gaming. I’ve configured the windows VM to automatically start Steam Big Picture, and from there, I just select a game to play with the Xbox controller, and the game launches, and runs with near native performance. The media centre keyboard/mouse is mainly just to be able to click on dialog boxes and the such. So far, I’ve successfully gotten the following games to run with zero extra effort:

• Red Dead Redemption 2 + a bunch of mods
• Cyberpunk 2077
• Portal 2
• Jackbox Games (I mean, these run on a potato, but it’s still nice!)
• Far Cry 6

I’ve not really used it for any multiplayer games - and one form of trouble I expect in the future is bullshit DRM causing problems. Games that use anti-cheat I am guessing are not going to be too happy running in a virtualised environment, so I wouldn’t necessarily recommend this setup for that - but if you mostly play single player campaigns - or you’re willing to engage in a bit of… ✨Creativity✨ in how you get your games running - you should have few to no problems.

The benefits of this setup are great. There is absolutely no console to have to hide in my living room and no screaming fan to enjoy the sound of. It’s like Google Stadia, except without the latency, without the quality penalty, and most importantly, it still exists.

One slight pain point - This is a HUNGRY vm. You’re going to want to have plenty of RAM to throw at it, and you’ll want more than if you just had a Windows install on the machine natively, since you’ll need some for Proxmox, and some for all the other stuff you’ll do with Proxmox. I went with 32GB - and give 16GB to Windows. Even this feels cramped some days.

Another slight pain point - You need some way to start and stop the VM! I came up with what I thought was a rather nice solution for this. My wife and I both use Telegram, and Telegram bots are an absolute joy to write. I wrote a simple Go service that runs directly on the Proxmox host (managed by Systemd), that spins up a web server. When it receives a request with a ✨Super Secure Secret Key✨, it executes qm start <vmid> - which starts the virtual machine. I then wrote a Telegram bot which only speaks to my wife and I (using Telegram as the authentication mechanism). If it receives a command to start the VM, it starts the VM. This Telegram bot of course lives on my Proxmox node inside a container, and therefore is able to make a web request to that Go service running on the Proxmox host I mentioned earlier. You could write these as one service for sure, but my Telegram bot does a lot more than just this for me, hence the separation. I haven’t shared the code for these yet because it’s terrible, it’s really nothing special.

This setup means that I can sit on my couch, fire up a telegram chat, and press a button, and about a minute later I am sitting in front of the Steam Big Picture UI, ready to play some games. I then use the Xbox controller to navigate the Big Picture UI, and I game. It’s probably as close to a console experience as one could have.

Final pain point - If you shut down the Windows VM, the GPU gets left in a really weird high power state, and produces a lot of heat considering it is literally doing nothing. It actually seems to run cooler while the VM is running. I did not want to rely on leaving a Windows VM running all the time, so I created the worlds most sad Linux VM. It is a really stripped down Ubuntu VM, configured exactly the same as my Windows VM, except it is given 1GB of ram and 1 core, as well as having absolutely no network access. It runs no services, it hosts no websites, it plays no games, it doesn’t even have the ability to be accessed in any way. Its only task is to hold the GPU. That same Go server I described above actually constantly polls the status of the windows VM with qm status <vmid>, and if it isn’t running, it’ll start the GPU holder VM. That then spins up, initialises the GPU, and does absolutely nothing else. It sits there, lonely, and sad. When I request the Windows VM start, the GPU holder VM unceremoniously gets stopped with qm stop <vmid>, and the windows VM gets started. A neat solution that keeps the power consumption of the GPU as low as possible. Better solutions to this are definitely possible, but I like my hacky solution.

How to set this up:

I had no idea before I set this up whether it would work or not (see my previous post: Home Lab Upgrades: Why This Mutant Motherboard/CPU Combo Could Be the Perfect Solution). There’s no technical reason why GPU passthrough shouldn’t work. It’s a relatively well established technology these days. It can be finnicky, but especially on the AMD GPU side things have worked pretty well for a while, bar some strange hardware bugs affecting earlier AMD graphics cards. However, as I discussed in my previous post, the platform I picked is anything but typical. It is a CPU that should never have seen a consumer, strapped to a motherboard that would prefer to not exist, running a hacked bios holding it all together, sold on Aliexpress at an extremely discounted price. The odds definitely needed to be in my favour for everything to hold together well enough for this to work. I already had an AMD Radeon RX-6700xt from my main desktop, so I decided to test with that.

BIOS Settings

After installing the GPU, I entered the BIOS of the Erying board, in order to ensure the settings were set up as needed. These are the settings that you’ll need to tweak, should you decide to do this. Do this before you attempt to set anything up.

• Advanced > CPU Configuration > Enable Intel (VMX) Virtualisation Technology - This is what allows Proxmox to do virtualisation on your CPU
• Advanced > Graphics Configuration > Enable VT-D (This is essentially IOMMU - tech that lets you virtualise hardware like your GPU)
• Advanced > Graphics Configuration > Disable Internal Graphics (Just to keep things simple, you can re-enable it later!)
• Advanced > Graphics Configuration > Enable Above 4GB MMIO BIOS assignment
• Advanced > PCI Subsystem Settings > Enable Above 4G Decoding
• Advanced > PCI Subsystem Settings > Disable Re-Size BAR Support (This caused crashing for me)

Proxmox initial setup

When you boot into proxmox next, it will boot and display a shell on your GPU. We don’t want this. We want Proxmox to ignore the GPU completely, so that a virtual machine can initialise it. To this end, ssh into your Proxmox node for the next steps. I should note for these steps I am heavily summarizing the excellent guide you can find here: Proxmox PCI Passthrough, and here: cjalas: The Ultimate Beginner’s Guide to GPU Passthrough - taking only the steps I needed.

Firstly, you’ll want to modify /etc/default/grub, specifically the line starting GRUB_CMDLINE_LINUX_DEFAULT. Add it if it doesn’t exist.

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefb:off video=vesafb:off video=efifb:off video=vesa:off disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,snd_hda_intel,snd_hda_codec_hdmi,i915"

Then, run update-grub.

You’ll then want to add the following lines to your /etc/modules file:

# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Next, you need to blacklist your GPU. This means that Proxmox will no longer attempt to initialise your GPU, and it will effectively become useless until we pass it to a VM.

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

Next, you want to add your GPU to VFIO. Run lspci -v and hunt for the output that relates to your GPU. Here’s what it looks like for mine:

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c5) (prog-if 00 [VGA controller])
        Subsystem: Sapphire Technology Limited Sapphire Radeon RX 6700
        Flags: bus master, fast devsel, latency 0, IRQ 139, IOMMU group 17
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        Memory at 4010000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 4000 [size=256]
        Memory at 50000000 (32-bit, non-prefetchable) [size=1M]
        Expansion ROM at 50100000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] Physical Resizable BAR
        Capabilities: [240] Power Budgeting <?>
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2a0] Access Control Services
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [410] Physical Layer 16.0 GT/s <?>
        Capabilities: [440] Lane Margining at the Receiver <?>
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

Note the numbers in the beginning, 03:00.0 in my case. You’ll next want to run lspci -n -s 03:00, in order to determine the vendor ID for the GPU. It may output two - one will be the GPU, the second will be the audio subsystem of your GPU. You’ll want to create the file /etc/modprobe.d/vfio.conf with these IDs, like so

root@kevs-server:/etc/modprobe.d# cat vfio.conf 
options vfio-pci ids=1002:73df,1002:ab28 disable_vga=1

Then run update-initramfs -u.

Finally, fully reboot your system, and check the monitor you’ve got attached to your system. As soon as the BIOS finishes loading, you should be left with a screen that says the following

Loading Linux <whatever version of the kernel> ...
Loading initial ramdisk ...
_

If so, congratulations, you’re ready to start configuring your VM!

Proxmox VM setup

It’s time to create your VM! Go ahead and hit the Create VM button in Proxmox. Don’t start the VM until the guide suggests you should. Start off by giving it a nice name.

• OS: select your OS. I had a Windows 11 ISO, so I selected that. Ensure you also select Guest OS Type Microsoft Windows.
• System: Machine should be set to q35, Bios to OVMF (UEFI). Tick the Add EFI Disk option and select somewhere to store the disk. Tick the Add TPM box and select the disk to store your TPM on.
• Disks: This is up to you, but I went with SATA, but you probably want VirtIO SCSI for better performance. If you’re running on an SSD, tick the SSD emulation box, and if your SSD supports Discard (It probably does!), tick the Discard box.
• CPU: Again, up to you, but I recommend giving it virtually all the cores in your system - and for Type you probably want to select Host, although any will work.
• Memory: Up to you, but I wouldn’t recommend Ballooning unless you’re really tight on RAM, given this a VM where you’re going to want as much performance as possible. Ballooning just means the VM can request more RAM and get it, but it requires further setup. I didn’t enable this.
• Network: Up to you

Next, it’s time to pass your graphics card through to the VM. To do this, select your newly created VM from the left in Proxmox, and select the Hardware tab. Click the Add button, and select PCI Device. Select Raw Device, and select your GPU. Tick All Functions, Primary GPU, PCI-Express, and Rom Bar.

You are also free at this point to pass through some USB ports to your soon to be running Windows VM. I for example pass through a wireless keyboard/mouse dongle, and an Xbox controller dongle. Plug these into your machine if you haven’t already, then select Add, and USB Device. I then suggest ticking Use USB Port, and selecting the ports they’re plugged in to. This means you can in future plug other stuff into these ports and it’ll just work. You can add more later, so no worries here.

Once you’re done, you should end up with a setup that looks something like this:

And finally, it is time to start the VM, and install Windows. You’ll need to do it from a monitor you attach to the GPU, as well as with a keyboard/mouse you passed through to the VM earlier. I’m sure you don’t need help there!

The experience of others has not been as smooth as mine, so be warned should you follow my path here. People with literally identical setups to mine have failed - seemingly because some of the engineering sample CPUs are capable of it, whereas others are not. Real PC hardware will probably work more reliably.

What is an Engineering Sample? Mutant?

It turns out, CPU manufacturers like Intel churn out a fair few “Engineering Sample” CPUs, which they then distribute to OEMs and ISVs so that they can test their hardware or software against some soon to be released CPU. For whatever reason, there appears to be a glut of these Engineering Sample CPUs from Intels 11th and 12th generation CPU product line, and they’re now finding their way into various strange products on Aliexpress, like this motherboard. I’ve also seen entire ‘Mac Mini’ style computers on Aliexpress that claim to use these same chips.

These boards have been nicknamed “mutants” in the communities which already knew about them. I’ve heard that these strange mutant boards usually find their way into countries where consumers cannot afford or do not have access to “normal” parts. I’ve seen reviews of this board from Brazil and Russia, implying they’re somewhat popular there. They’re not a new concept as I thought, and have been seen before.

Why might we want such a motherboard, though? For starters, the price is pretty fantastic. For 220$, we get a board which features the following:

• A CPU equivalent to an Intel i7-11800h (8 cores, 16 threads, turbo frequency of 4.60 GHz, TDP of 35w)
• 4x SATA ports
• PCI-Express x16 slot, PCI-Express x1 slot
• 1Gbit Ethernet
• A NVME slot

For 220 CAD, you’d be hard pressed to find a CPU/Motherboard combination that performs as well as this board should. It’s a great deal, theoretically.

Having a destined-for-mobile CPU is nice for other reasons too. It’s low power! These mobile chips are optimised to run as fast as possible, in as small a power envelope as possible. For a home server use case, this is a desirable property, since this will lead to lower bills, and less heat to dissipate.

Why would I want one

? For the past few years, I’ve been hosting my own services using a ‘home lab’ setup, which has gone through several evolutions. I started with an old netbook, then moved to Raspberry Pis, working my way through the generations of Pi, all the way to the Pi 4 8GB. I then added a consumer grade NAS to my setup, which I featured in my article about replacing Google Photos with a self hosted alternative.

After a few months with the Pi 4, I got tired of managing an ARM machine. This isn’t a slight against the Raspberry Pi. They’re amazing little boards and are fantastic for a lot of use cases - but they’re not perfect. Not all software is ready for ARM, and the hardware can be a little flaky if you use the wrong power supply. That flakiness can corrupt your MicroSD card, leaving you offline.

So, I decided to level up to a Tiny PC - the Lenovo M710q. These Tiny PCs are available for fairly cheap online second hand. I used this machine in my article about using OCR to extract data from a set of Eufy weighing scales. This machine and NAS setup satisfied my needs for around a year, but eventually, I decided I wanted more.

For a lot of people who are considering how far to take their own home lab, I’d suggest stopping here. This setup was mostly great for me! At the time, I had a fairly average internet connection (80mbit down, 20mbit up), and this setup managed to keep up with that fairly well. It was enough to host all the home services I wanted to host. If you were to purchase, say, a Synology NAS, instead of a Terramaster one, it would probably be fine. My Terramaster NAS however began to worry me. Users including myself were regularly finding security holes in the NAS, but the real cherry in the icing was discovering data corruption due to my choosing to format my drives as BTRFS. Turns out, there was some nasty bug in the Terramaster firmware which lead to data corruption for users who picked BTRFS. Terramaster of course provided no migration path away from BTRFS to EXT4 (the other supported filesystem) when they quietly announced that newer versions of this NAS firmware would stop offering BTRFS as an option, so I decided enough was enough. No more rinky dink NAS for me.

I have since moved to Canada, and now have significantly faster internet. This is where the shortcomings of mounting a NAS SMB share on another machine became apparent to me. For no reason at all, the NAS would entirely lock up when downloading stuff from the internet at more than 50MB/s. Transfers would just stop occasionally, and resume about 30 seconds later. Further comically awful security flaws in TOS were discovered, so I finally decided to action what I’d previously decided, and got rid of the NAS + the mini machine. I started researching options for something better.

Enter Erying

I had some features I really wanted to try to get on the new home server. These were:

• Host a larger number of drives than 2, as my current 4TB mirrored array is getting pretty full
• Be x86 - They’re just easier to manage than ARM boards, in my opinion
• Easily handle gigabit network traffic.
• Offer the ability to go higher than Gigabit in future
• Run Proxmox
• (Stretch) GPU Passthrough to a virtual machine

I’d been thinking about this project for a little while, but finally decided to go ahead and do it when I spotted this Erying board. It ticked all of these boxes exceptt he one regarding GPU Passthrough. There were reports that some users (including CraftComputing) were unable to get this working. That was not an absolute necessity for me, so I took the plunge anyway. I placed the order, and anxiously awaited my motherboard.

The board arrived. A little beat up from delivery, but still mostly structurally intact. Hopefully, what’s inside survived!

The back of the box makes a bunch of claims about the motherboard. I particularly enjoy the “Anti-thunder Design (double protection)”. Sounds extremely dramatic! Let’s open it up and take a look!

As promised, there’s what looks like a motherboard, an IO shield, a SATA cable, and that’s it! I didn’t even get a manual. Let’s take a closer look at the board.

Interesting! We have a somewhat unusual setup here around the CPU, where there appears to be a copper shim between the world and the CPU die itself. This’ll be because mobile CPUs don’t usually have Integrated Heat Spreaders like their Desktop CPU bretherenm and silicon dies are very fragile. There’s also a metal plate around the copper shim to hold the it to the die at some pressure. The mounting holes for the CPU cooler appear to be LGA17XX style, so most CPU coolers will fit to the board with no issue. Nothing special is required in this department, given the low power usage of this CPU.

Other points that appear interesting to me are the extremely weedy VRM heatsinks surrounding the CPU socket. We’ll measure those later. Other than this, everything looks in order.

The IO is basic, but it’s all there. You’ve got your four USB 3 ports, two USB 2 ports, a Gigabit Ethernet port, 2x Displayport + 1x HDMI so you can use the onboard GPU, if you wish (this CPU features Intel HD 630 graphics). There’s also audio jacks.

The build

I decided to throw it into a spare case I had, and upgraded some components in my main desktop to free up some parts for this build. It ended up being the following:

• DEEPCOOL Macube 100 case
• DEEPCOOL Gammaxx 400S CPU Cooler with 4-Heatpipes (overkill, but the cheapest at the time!)
• 32GB Corsair Vengeance 3200MT/s RAM
• Erying motherboard
• Intel ES 0000 CPU which is pretty much an 11800h
• Sapphire AMD Radeon 6700xt GPU
• Sabrent Rocket 1TB NVME SSD
• 2x Toshiba N300 4TB Hard Drives
• Seasonic Core GC-500w PSU

The lack of manual posed a bit of a challenge, since I didn’t know where to connect the power switch! They hadn’t labelled the ports on the motherboard, nor was there any documentation on the internet I could find. Thankfully /r/EryingMotherboard came to my rescue, and I was able to continue.

After I hooked it up to my TV for testing, I was somewhat surprised to see it boot! Then, I remembered that the SSD I used already had an OpenSuSE Tumbleweed install present, since I’d migrated it from this SSD to a new one, so naturally it immediately booted to that.

Now I’d confirmed the board worked, it was time to set it up properly, and get it into service!

The Scary BIOS update

The seller offered me a “more powerful” BIOS, but I had no idea what “more powerful” meant exactly. When I asked for further clarification, the seller directed me to a Microsoft Onedrive link that seemed a little sketchy, but I decided to take a risk and download it. The ZIP file contained instructions to flash an included BIOS, but they were in Chinese. I was able to translate them with the help of ChatGPT (how topical!). I followed the instructions and was presented with a screen of scrolling text that lasted about 10 minutes. Eventually, the board rebooted, and I was relieved to see that the BIOS had been successfully updated. It was a bit of a nerve-wracking experience, but I’m glad it worked out in the end.

Let’s take a look at how the BIOS looked before and after the flashing process. Let’s first look at the before photo:

As yes, old familiar. Blue borders with large grey areas with black text. Incomprehensible settings. Press F10 to Save and Exit. Everybody who has tooled around with a BIOS has probably seen this one. How quaint. Now let’s take a look at the “more powerful” BIOS.

Whoa! Now there’s LIGHTNING in the background that makes portions of text unreadable! That IS more powerful looking! I Love it. As to what other changes were made, I have no idea. Oh well!

Bios configuration

Through tooling around in the BIOS, I was able to get my RAM to run at a maximum of 2933 MT/s, but no higher. For some reason, this board would NOT let me go to 3200 MT/s no matter how much I messed around with memory timings, but whatever, no big deal. I also enabled Intel VT-d, Intel Virtualisation, and IOMMU - all technologies related to virtualisation that are essential for what I’m about to do with this system. I did not however enable Resizable BAR, since this caused issues which I’ll elaborate on later.

Proxmox

Next, I installed Proxmox on the system, but I’m going to leave that fun for a second part to this blog post. See you then!

• Huaweis’ weirdo Supercharge standard
• USB PD 2.0 and 3.0
• Qualcomm Quickcharge 2.0, 3.0, and 4.0
• Programmable Power Supply (PPS) charging as used by various Samsung phones and laptops
• Adaptive Fast Charging (AFC) as used by various other Samsung devices
• Super Fast Charging (SFC) as used by various other more different Samsung devices,

Every single device I’ve plugged into this absolute unit of a power supply has charged as fast as I’ve ever seen it charge, and its 3 USB C ports can even charge multiple laptops. Buy one from this here affiliate link. Highly recommended.

I decided I’d let the phone drain to roughly 10%, and then not touch it for a full charging session. It was placed in a room that is held pretty steadily at 22 degrees centigrade. I should also note this Pixel 6 Pro is only around 8 months old, but I’d estimate the battery is a little more worn than usual due to some extremely heavy use it sustained while it was my only connection to the internet.

I quickly whipped up a terrible shell script that used adb wirelessly to log out the values. I can’t take all the credit for the terribleness, it was actually authored by Chat GPT - my new favourite tool for quickly hacking up prototypes of terrible shell scripts. When it does the terribleness, I feel less bad about it!

#!/bin/bash

while true; do
  timestamp=$(date +%s)
  adb shell dumpsys battery | grep -Eo '[0-9]+' | tr '\n' ',' >> battery.csv
  echo "$timestamp" >> battery.csv
  echo >> battery.csv
  sleep 30
done

One complete charging session later, I wished I’d sampled more frequently than every 30 seconds, but the resolution was high enough to observe a couple interesting things. Firstly, a charge from 11% to 100% reported took 1 Hour, 43 Minutes and 35 Seconds. Internet reports suggest 2 hours for a full charge, so this seems to indicate it is charging as fast as it can.

Firstly, I decided to take a look at how the self reported charge level (in percent) compared to the temperature the battery reported.

This graph showed something I was expecting. We can see at around 50% charge, the temperature of the battery peaks at around 34.7c. Then, the charging speed drops a bit, and the temperature drops.

Next up, I was curious how the battery voltage changed over time. I decided to compare it against the self reported level.

Here, again, when the battery hits 50%, the battery voltage drops precipitously, indicating a drop in charging power. It then steadily rises to its observed maximum of 4.42v, and stops, indicating the cell is full. Another interesting detail we can see in this graph is the cells voltage keeps rising even as the phone reports itself 100% full, indicating that the phones own estimations were slightly off. I decided to dig deeper.

Next I graphed the mAh figure that the phone exposes against its voltage.

With Li-ion cells such as the one we find inside the Pixel 6 Pro, we expect to observe a fairly steady voltage through a significant portion of the charge cycle, as these cells try their best to hold a high voltage for as long as possible before rapidly dropping when the cell is almost empty. This is only true when they’re charged with a relatively constant current. Modern phones do not charge with a constant current however, and vary it in various ways to try to preserve the health of the battery and to control their heat output. We can however observe this behavior somewhat when the battery cell voltage hits roughly 4400mV.

Indeed, the phone did keep charging when its own battery gauge reported 100%, gaining a whole 144mAh of charge, and the battery voltage rising another 6mV. It’s hard to know exactly why this is, but it could just be a case of the phone trying to encourage you to remove the phone from the charger slightly below 100% to avoid extra battery wear, or just the charging gauge calibration being slightly off. I also should note that the phones “adaptive charging” options were enabled, which assumedly will have some effect on charging. I will perhaps perform this test again with these options disabled in future to see how that affects the charge curve.

To really see this, I graphed battery level against battery mAh.

Sure enough, we can see this behavior here too. If you’re interested in the raw data, you can find it here.

Several years ago, my wife and I decided to immigrate to Canada and applied for the Express Entry program, which allows those with certain qualifications to obtain permanent residence in the country. We anticipated the process to take between six and twelve months. However, the pandemic caused Express Entry to be effectively halted and we eventually gave up on the process. I then periodically would look into alternative routes in.

I applied for various jobs in Toronto and eventually found an employer who was willing to hire me, even though I planned to relocate. The work permit for this position was supposed to be processed through the “Global Talent Stream,” which typically takes only two weeks, but due to the pandemic and other global issues, we were told to expect a processing time of two to eight weeks. I decided to move out of my rental apartment in the UK after four weeks and stay in an Airbnb if the visa took the full eight weeks. However, the visa ended up taking 17 weeks and during that time, I had a number of unexpected experiences and learned valuable lessons.

My experiences living in hotels.

After four weeks of waiting, we were forced to move out of our apartment and into a hotel. In selecting a place to stay, our priority was to find a location that would allow us to bring our pet cat and was as affordable as possible. We wanted to save our money for furnishing our future apartment in Toronto.

We initially stayed at a budget hotel chain (which I will not name) as a temporary place to stay while traveling. We just needed a place to lodge while travelling. The experience was terrible. The mattress had a large urine stain, there was a bloody handprint (actual blood) on the ceiling, and there was no hot water for multiple days. As a result, breakfast was not served on several days. One night, the elevator near our room was constantly alarming and kept us awake. The next morning, one of the two elevators had an “out of order” sign on it, but when I tried to use the other one, the “out of order” lift showed up and the sign fell off. I took the stairs.

The straw that forced us to change hotels however was that one night a young girl attempted to walk into my locked room at 9pm, while I was undressed. It was 40c on that day, with no AC, hence my state of undress. She succeeded in opening the door, seemingly having a valid key card for it. Thankfully my wife was fast enough to quickly get to the door and stop the child from walking in and seeing anything. When we got to reception to try to figure out why some other family was trying to occupy our room, reception didn’t even seem surprised, apologized, and that was that. After experiencing the issues described above, and going round and round in circles with reception, we decided we would leave the hotel that night. The hotel eventually refunded us for the three nights we didn’t have hot water from our our 16-night stay, but refused to refund the final four days of our booked stay, claiming that the issues we experienced were not serious enough to warrant a refund.

We moved to a second hotel with this same chain, which was far better. Nothing much to note there, apart from at this point the reality of living in a hotel had set in. There are no cooking facilities, so I pretty much just lived on supermarket meal deals, and stuff that could be cooked with nothing other than hot water. Luckily, this particular hotel was near to a friends place however, and I am forever thankful for the meals he cooked us, and allowing us to hang at his whenever we needed some brief respite. Thanks Pete, you helped keep me sane and scurvy free. We stayed here for approximately 3 weeks.

Every day without fail, I’d be calling a number I found online to get updates on the status of my visa. At one point, I wrote an autodialer where I could press a button, and it would navigate through the Canadian Immigration phone tree and alert me when it got to the point where the update would be delivered. The estimate had now been updated to 12 weeks, so we assumed that we only needed one more hotel stay, and we would finally be on our way to Canada.

Our finances really were taking a beating here too. I was lucky enough to have been able to save up a decent chunk of cash through the pandemic, but at approximately 1000 GBP per week, I was fast running low, so we had to find a cheaper option.

We decided to stay at an Airbnb in London because we thought it would be more cost-effective to cook our own meals rather than eating out at restaurants. However, our experience ended up being a disaster. The Airbnb was a converted house with 18 tiny units, and the entrance was a broken gate leading to a garbage-filled garden that reeked of a strong smell. Upon entry to the building, there was a lovely sign written by one of the other occupants complaining of the constant parcel theft issues that were apparently rampant in that building. The kitchen was so small that it was barely larger than a shower stall, and there was no room for a microwave, so it was placed on a bedside table without any nearby outlets to use it. Despite these challenges, my wife was able to make the most of the situation and we were able to eat well. I am SO grateful to her for her resilience in such a difficult and unpleasant environment.

After our stay at the previous location ended and it became apparent that the 12-week estimate was inaccurate, we moved to a more expensive place. The increased cost was partly due to the sudden increase in demand for accommodation in London following the death of Queen Elizabeth II. We were fortunate to find a slightly more expensive location that had not yet adjusted their prices and stayed there for two weeks. Unfortunately, we were unable to extend our stay because the place was fully booked, so we had to move again.

Our final hotel stay was at a budget brand of a more upscale hotel located near Heathrow Airport, which made it very affordable. While I eventually became accustomed to the constant smell of jet fuel, this hotel was one of the most difficult to stay at. This wasn’t due to the hotel really, it was fine. It was in the middle of nowhere making eating difficult, and it was intensely boring, but the real problem was our patience had just run out at this point. They overcharged us 1000 GBP and it took me around three months to resolve the issue with my bank after disputing the charge. As I had already maxed out my credit, this had a greater impact on me than you might expect. Thanks, hotel! On top of this, there was still not a clear idea as to when the visa would be issued, meaning the impending homelessness was feeling more and more possible.

We eventually got the visa, and departed a few days later, landing in Toronto with very little money to our name. Thankfully, things are slowly sorting themselves out now both I and my wife are bringing in a salary, but it’ll be while before we’re back at 0.

Lessons learned

I learned a few important lessons during my experience with short-term accommodation. First, never trust a visa estimate and don’t make any plans until you have the actual visa in hand. This mistake was entirely my own and I regret it deeply. Second, living in a hotel is not enjoyable. Being unable to cook or clean for oneself is miserable. Third, Airbnb is no longer a reliable option for reasonable, short-term accommodation. Even via other avenues, most landlords are hesitant to rent out their properties for a short, undefined period. Fourth, I learned that it is not advisable to rely solely on a smartphone for technology, as I was without a computer for the first eight weeks and had to purchase a laptop. Finally, it is important to be prepared for unexpected events, such as the death of a monarch, which can have an impact on the availability and cost of accommodation.

Also worth noting is a lot of this article was written by OpenGPT, for fun.

I received a message concerning a computer case I was attempting to sell on Gumtree. I’d listed it about 15 minutes prior to the message arriving.

Note: If you’d prefer to read this as a series of screenshots, go ahead.

Scammer: I was interested in computer case

Scammer: Good afternoon, I’m about an announcement on gumtree

Scammer: Is it convenient for you to speak now?

No suspicions yet, seems legit!

Me: Hi, sure, are you interested?

Scammer: is he in good condition?

Me: Particularly good condition yes, it was used for around a year but just lived on my desk. There’s a few scuffs here and there but nothing particularly noticeable. All screws/fittings that were included are still there, only thing that’s missing are the shields where you’ll probably be putting a graphics card anyway.

Scammer: okay, are there fans included?

Me: No, it doesn’t come with any fans

Me: It can fit a 120mm or a 140mm fan

I decided to be extra helpful and link him to a LinusTechTips video reviewing this computer case. Can’t say my customer service isn’t on point. At least he wasn’t questioning whether I was Romanian or not!

Me: https://youtu.be/8ptbpKpObzs this is the case, if you’re curious.

Scammer: well, for such a price everything suits me

Scammer: when can you pick it up?

Wait… When can I pick it up? Strange, although it could have been a language barrier thing. Whatever.

Scammer: or will you send it by mail?

Me: Pickup only unfortunately, not cost effective to send computer cases by post!

Me: You can pick up anytime today from Mayfair - just let me know when to expect you

Scammer: I’m at work today, I’ll be free only next weekend

Scammer: Can I pay for the item now and pick it up next weekend?

Me: That would be fine yes! I’ll hold it for you. You want to pay via PayPal/bank transfer?

Me: You can also pick it up tomorrow, or any day after Friday

Me: It’s up to you

Scammer: It will be convenient for me to pick up the food next Saturday

Pick up… The food? I’m not sure who eats computer cases, but I’m not sure I want to cross them! Still I suspected nothing so far.

Scammer: yes, I can transfer to the card and paypal

Scammer: as you prefer?

Me: https://monzo.me/kevinnorman/10.00?d=Gib%20me%20money%20for%20writing%20article

I sent him a link requesting money as generated by my bank. That’s a real link, by the way, feel free to send me money, I certainly would be grateful!

Given this is an ITX case (a small one which only fits certain hardware), and because I’m a nice guy, I thought I’d ask him what he’s putting inside of it to double check he was aware that the case was an ITX case.

Me: Out of interest, what are you going to put inside it?

Scammer: Intel core i7 and gtx 2070

Scammer: or it is possible through PayPal, I have money there

Me: Sure, kn100+paypal@kn100.me

Scammer: more precisely core i7 9700k

Me: Very nice

Me: Will be a very powerful tiny computer!

Scammer: Yes

Scammer: I will send a check, do you mind?

At this point my scam senses started tingling. A check? Does he mean a cheque? Does he mean a security check? For a £10 case?

Me: I’m not sure I understand

Scammer: one second

Scammer: https://paypal DOT 3sds DOT site/pay/?id=49688501 (I have intentionally garbled this link. Visit at your own risk.)

Scammer: this is check

Scammer: fill in the data to receive money

Scammer: understand?

Ah. A Scammer. Visiting this page took me to a page that looked like the following. An obvious scam, but it’s pretty reasonable to expect that a lot of people might fall for this.

I started to consider what to do, so I stalled a bit by just telling him the link didn’t work. In this time I would quickly spin up a virtual machine, and do a bit of recon.

Me: It’s not working

Scammer: what does not work?

Me: That link

Me: It just 404s

Scammer: One second

Scammer: https://paypal DOT 3sds DOT site/pay/?id=49688501

Me: Give me a minute my dog started screaming

Scammer: Ok

Scammer: try to copy the link and paste it into the browser

Still trying to stall for time, I try to delay by asking more PC related questions to waste his time and to give me time to think.

Me: What power supply are you planning to use?

Scammer: Corsair 750w

Me: The link still isn’t loading, but now I’m getting error 509

Me: 500*

I decided to claim a different error message was occurring, really get him worried. I was honestly stalling for time.

Scammer: I’ll write to technical support now

Scammer: https://paypal DOT 3sds DOT site/paychekoff/?id=49688501

Scammer: they stuck this link

Scammer: they said everything is fine

Ah, a nugget of information! With a lot of these scams, the scammer you speak to does not manage the tool they use to scam you. The tool is purchased from some skiddie, and is then run by the scammer. The use of the word “they” in his messages above led me to believe that he was in direct contact with the person responsible for hosting the tool they were attempting to use to scam me.

At this point, I decided to fire up an image editing tool.

Me: Okay now it’s refusing to accept my details

Scammer: try with another browser

Me: It says error 418 now… PayPal is really unreliable today maybe we could try another way?

Scammer: I only keep money on paypal

Me: It loaded this time at least, but I’ve tried submitting like 5 times

Me: Maybe you can withdraw 10 pounds and bank transfer me

Scammer: send a screenshot of the error, I will send to those support

Me: One second

Scammer: Well?

So I edited an image with an error 418. For the uninitiated, according to Mozilla, Error 418 “indicates that the server refuses to brew coffee because it is, permanently, a teapot.”. I used this error for two reasons. Firstly because it was hilarious to me, and secondly any person who knows anything about web development is very likely to know this error and realise what is going on. If this person catches on, I was unlikely to be able to have much fun with them anyway.

Scammer: if it doesn’t work now, write to those support, at the bottom right

Scammer: it just works for me

It just works for you huh? You’re just paying yourself money? Seems legit Mr. Scammer!

Me: I tried writing to the chat but it seems its gone offline

Me: also, the payment boxes have disappeared completely now, maybe too many attempts?

Scammer: Maybe

Scammer: all this is strange

To really confuse this guy, I decided I’d edit the screenshot so the online chat appeared offline. You’ll notice the little circle on the chat thing turned from green to red. I also hid the ‘payment’ boxes, as promised. What I wouldn’t give to see the conversation between this guy and the tech guy running it.

I wanted to create a sense of urgency, and to force this scammer to regenerate their payment link, so I engineered a situation where he’d increase the price.

Me: I think I might have another buyer for the case, I might see if they can pay for it in a different way, I just want shot of it

Scammer: as you wish

Scammer: technical support wrote: the server was restored

Scammer: can you try again?

Me: Hmnn, this other buyer is offering me cash today. Perhaps if you want it we can say 15 pounds?

Scammer: okay let’s go for 15

Me: The link still says 10

Scammer: now we will create a new translation

Scammer: https://paypal DOT 3sds DOT site/paychekoff/?id=49688501

Hey, a new link, with a name! Thanks Chekoff!

Scammer: 15£

Scammer: works?

Me: I don’t know what the hell is going on

Now, I decided to give the Script kiddie behind this nightmares. I edited the payment page to read NaN for the payment amount.

Me: this says: To Receive: NaN

Scammer: I do not know what’s happening

Scammer: try to go through another browser

Scammer: I’m already so tired, I’m ready to give 20 £

Next, I decided to engage with the scammer via their “tech support”. This screenshot is undoctored.

Scammer: how will tech support answer write me

Hilariously, after I refused to provide my card details via chat, the scammer messaged me again on Whatsapp to ask why I wasn’t providing the card details!

Me: I have no idea but I am not typing my card details into a text box for 20 quid

Me: oh man I just went to reply to support and got this

Next, I threw him a curveball. I edited the screenshot of the chat to have a fake warning indicating that the “free trial period” had expired. Next, I asked him if he’d like to sign up to my Onlyfans. I also linked him to a nursery rhyme for no good reason.

Me: Here’s a weird idea… I actually have an Onlyfans, and Onlyfans can be paid for with Paypal. What if you sign up to my Onlyfans and I make a tier that is 20 pounds?

Me: https://www.youtube.com/watch?v=B6en-O5yF0o

Me: oh sorry that link wasn’t for you

Here, he went quiet. I decided a few hours later to try to re-engage him by letting him know other buyers had lost interest. What he didn’t know is I’d already sold the case to someone else.

Me: this other buyer has fallen through, so it’s just you now. Not sure what to do.

Scammer: how paypal works, i will transfer you £ 20

Scammer: Ok?

Me: how?

Me: Hello? Do you still want it?

Scammer: Yes

Me: Then how shall we conduct this business transaction sir

Me: I’ve got the case, you’ve got the cash, can I make it any more obvious?

Scammer: I’m going to try now

Scammer: try now

Scammer: https://paypal DOT wis3 DOT site/pay/?id=73531466

Ha! He’s given me another domain to report. Excellent. More sleuthing indicates both domains were registered with the same email. A tiny bit more looking showed that there were around 50 of these domains all serving the same content. Reg.ru of course got a list.

Scammer: work?

Another curveball. My sleuthing through WHOIS records lead me to believe the scammer in question was Russian too, and the domain was registered with Reg.ru. I decided to mock up a fake domain suspension notice. The Russian reads “This website has been blocked. Get in contact with the support team”.

Scammer: I don’t know, try using another browser

Me: sorry let me try on a computer, this is tiresome

Me: Same thing

Of course, the same thing on the desktop! Left a nice easter egg in the tab bar though, which wasn’t noticed of course.

Scammer: so is the deal canceled?

Me: I can’t seem to get payment from you so I’m not sure what you want me to do

Me: maybe you have other suggestions

Scammer: there is a suggestion to put an end to this

HE’S FINALLY CLOCKED ON!

Me: go onnnn

Scammer: Goodbye

At this point, I’d done a fair bit of sleuthing, and determined his name was probably not Mark. I’d also reported both domains to their registrars, reported abuse to their host, reported them to the various safe browsing blacklist services, etc.

Me: Have a lovely evening Mark :)

Me: Does your scam work often?

Scammer: almost always

Me: how much do you make?

Scammer: it’s a secret

Me: welp, I hope you find some joy in life that doesn’t involve destroying others lives, and I hope you enjoyed having so much of your time wasted, I certainly had a whale of a time in Photoshop :)

The concept is was simple. Sites which wished to have their content aggregated would publish an RSS feed - which was an XML document that had a standard set of tags to describe the content on the site in a standard fashion. You can see an example of an RSS feed right here. Many, many websites published RSS feeds through the years after the death of Netscape in 2003, and similarly many pieces of self hosted software that did not require users to make use of a website to view their RSS feeds, and for a while this seems to be the main consumption model for RSS. Tools like Mozilla Thunderbird, AOL Explorer, Claws Mail, IBM Lotus Notes (Which crazily was discontinued almost exactly 2 years ago, and saw its first release 33 years ago!), Firefox, Microsoft Outlook and many more all supported RSS feed syndication of one form or another.

Then came Google Reader in 2005. Those who love RSS generally speak fondly of Google Reader. It was a fairly long lived tool by Google standards - lasting 7 years, and eventually being killed in 2013. It is hard to estimate just how many people were using Google Reader, but competitor in the space Feedly announced that in the 2 weeks after Readers discontinuation over 3 million people joined.

What is interesting to note here is the flip-flopping between ‘cloud’ based readers, like my.netscape.com, and offline readers like Thunderbird. I’m not certain why this is, but RSS definitely lends itself very well to server side processing, especially in the days prior to the enormous slow React apps and the browser per application Electron apps of today. Power users were also probably more open to the idea of giving up the keys to who controls their data in the name of convenience with tools like Google Reader, but that is just a guess.

Also of note is the fact that RSS is very alive and well today. Feedly seems to be one of the largest feed readers in use by users today (at least based on the server logs for kn100.me). I can tell this because when I check my Nginx server logs, when Feedly requests my feed, their crawler reports the number of subscribers to my feed in its user agent via Feedly. This is a lovely feature, by the way!

Feedly and other cloud based RSS aggregators pose an interesting problem however, and therein lies the peril of RSS. Firstly, I need to quickly describe how my RSS feed comes to exist. I myself am not a user of RSS, but enough readers complained about the lack of a feed that I eventually added one. Hugo, the static site generator I use, can easily generate RSS feeds, if provided a template for doing so. Such templates look like this:

{{- $pctx := . -}}
{{- if .IsHome -}}{{ $pctx = .Site }}{{- end -}}
{{- $pages := slice -}}
{{- if or $.IsHome $.IsSection -}}
{{- $pages = $pctx.RegularPages -}}
{{- else -}}
{{- $pages = $pctx.Pages -}}
{{- end -}}
{{- $limit := .Site.Config.Services.RSS.Limit -}}
{{- if ge $limit 1 -}}
{{- $pages = $pages | first $limit -}}
{{- end -}}
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>{{ if eq  .Title  .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }}</title>
    <link>{{ .Permalink }}</link>
    <description>Recent content {{ if ne  .Title  .Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }}</description>
    <generator>Hugo -- gohugo.io</generator>{{ with .Site.LanguageCode }}
    <language>{{.}}</language>{{end}}{{ with .Site.Author.email }}
    <managingEditor>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</managingEditor>{{end}}{{ with .Site.Author.email }}
    <webMaster>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</webMaster>{{end}}{{ with .Site.Copyright }}
    <copyright>{{.}}</copyright>{{end}}{{ if not .Date.IsZero }}
    <lastBuildDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</lastBuildDate>{{ end }}
    {{ with .OutputFormats.Get "RSS" }}
      {{ printf "<atom:link href=%q rel=\"self\" type=%q />" .Permalink .MediaType | safeHTML }}
    {{ end }}
    {{ range $pages }}
    <item>
      <title>{{ .Title }}</title>
      <link>{{ .Permalink }}</link>
      <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ safeHTML "<![CDATA[" }}{{- .Content | safeHTML -}}{{ safeHTML "]]>" }}</description>
    </item>
    {{ end }}
  </channel>
</rss>

In this template, I expose my entire post content for all posts on my blog. The RSS feed output seems to work just fine, and those passionate RSS users I mentioned earlier were perfectly happy with this, until recently. I just rebuilt this blog from the ground up. I did this because when I first built kn100.me, I had very little idea what I was doing with Hugo, and the codebase (if you can call it that!) had gotten very messy, the CSS had gotten messed up, and it was getting more and more difficult to manage. I created a brand new Hugo blog, ported all my content across, reorganised all my static assets, ensured all the permalinks hadn’t changed, actually properly customised my templates, and was pretty pleased with the result. I deployed it, and all seemed well.

A few days later, another project of mine, the Cybiko Game Archive microservice I’d written to serve up a catalogue of games that existed for the Cybiko was reporting errors. Unfortunately, a Script Kiddie had found it, and was bombarding it with dumbass requests to /wp-admin.php, /admin, etc, and the code was dutifully doing exactly what I told it to, which was to log it. I didn’t really see the point of reworking this microservice to deal with this, but didn’t want to spend much time on it, and I didn’t want to risk turning logging off, so I decided to do something more drastic.

The Cybiko Game Archive service is fairly basic. I give it a directory full of content, and it creates an in memory index of it, which is then served using Go templates. The pages it produces do not change often, so are a good candidate for being static content, so I rewrote it to optionally export Markdown, which I could import into a new section of my blog, where Hugo would be responsible for generating the pages. The result of that work is what you now see on the site. This reduced the administrative overhead of hosting it, and meant that I could deploy it along with my blog, no problem.

This all went very well, and I ended up with a fairly acceptable result, which I later deployed. The Cybiko Archive sans a microservice was live, improving administrability, security and performance.

A few hours later, a reader tweeted at me asking why their RSS reader was bombarding them with literally hundreds of pieces of content that weren’t my usual kind of content. I investigated, and came across the fact that my RSS feed now listed every single Cybiko game, along with my regular posts. These Cybiko game posts were never intended to go into the RSS feed, and I incorrectly assumed that my RSS generation logic would not generate entries for posts that were not part of the main section of the blog.

This should be an easy fix, I thought, I’ll quickly check out the template for RSS generation and add a filter, so that it would only generate entries for pages that are part of the main section of my blog, called posts.

Find: {{ range $pages }}
Replace: {{ range (where .Site.Pages ".Section" "posts") }}

I regenerated the static pages, deployed them, and called it a night. The next day, another reader contacted me reporting exactly the same thing. And a third. And a fourth. I quickly figured out they were all using Feedly, and then later learned that while Feedly is quite happy to add whatever content you add to your RSS feed to their internal store, they do not remove content that you remove. This is fairly similar to what a lot of RSS reader software that the user installs on their computer does, however the cloudy nature of Feedly removes an important avenue of control from the user. With RSS reader software, it is possible to remove and re-add a feed, thereby grabbing a brand new copy of it. When a content creator like myself makes an error like this, users can fairly easily do this after the mistake had been corrected, mark everything below their most recently read post as read, and get on with their life. Not so with Feedly, since it does not appear they ever remove content that was removed from an RSS feed. It’s been a week since this issue was first reported to me, and there does not appear to be anything I can do about it.

I attempted contacting Feedly via Twitter and via Email, but have not heard back from them. Unfortunately, any Feedly users who were subscribed or now subscribe to my blog now just have to deal with it, and there is as far as I know absolutely nothing I can do about it from my end. I just have to apologize to my readers. Therein lies the peril of RSS.

The point I hope to make clear is that if you’ve never checked, your RSS readership is likely larger than you realise. It was certainly far larger than I realized. kn100.me is a tiny blog, and I’ve been enjoying the slow growth that being on the internet for a while brings. I do this for fun, and have made exactly no money from doing it. Losing a few RSS users is sad (and I’ve definitely seen a dip in my Feedly readership), but is not a disaster for me. If you live off of blogging, be very careful with your RSS feed. One mistake can permanently wreck your RSS feed for users of cloudy platforms you have no control over.

Thanks!

Unfortunately old, long term unused Cybiko Classics suffer from exploded batteries. Worse, the residue will eat away at the original battery contacts, and will even begin to damage the motherboard if left in for long enough.

Get the batteries that are in there out as soon as possible, and clean the battery contacts as well as the Cybikos own contacts as best you can. You may also consider disassembling the Cybiko to clean any residue that may have gotten to the motherboard. Do NOT mess with the LCD, the ribbon is incredibly flexible and I have killed a unit fiddling. You could attempt to reuse the batteries, but I would strongly suggest instead either finding replacements, or as a cheaper and worse alternative doing what I did and attaching an alternative power source. Two Ni-MH 1.2v AAA or AA batteries in a holder soldered to the battery contacts will work just fine. It is also STRONGLY recommended you do NOT solder directly to the batteries, and instead purchase a battery holder instead. Doing a poor job of soldering to the batteries at best can cause irreparable damage to the battery, and at worst can cause a catastrophic failure.

Hardware to connect to Cybiko Classic

If you do not have the serial cable bundled with the Cybiko, you should know that they’re practically impossible to find. If do not mind making a small after-market modification to the Cybiko, you will need to do some soldering work to add some wires, which we will later attach to an RS232 port. This is necessary because Cybiko used a proprietary connector and as far as I can tell no other device used this connector, and therefore the cables are becoming quite rare.

The Cybiko motherboard has a number of test points, and thankfully four of them are exactly what we need to solder to in order to bring the serial connection it uses to connectors we can actually connect to. See the diagram below which highlights each test point. I suggest using the thinnest insulated wire you can get your hands on, in order to make running them inside the Cybiko as easy as possible. One possible egress point for these wires would be the hole on the right hand side of the battery compartment. I also suggest at this point making a note of whichever colour you soldered to each test point, so once you reassemble the Cybiko, you know which is which.

TP47 is TXDL, which goes to the RXD of your RS232 adaptor, TP48 is RXDS, which goes to TXD of your serial adaptor, TP49 is RTS, and the black pin that I’ve stuck on the side of the reset switch is GND, because the outside of the reset switch is grounded, so you can just solder there, or anywhere else you can find ground." title=“TP47 is TXDL, which goes to the RXD of your RS232 adaptor, TP48 is RXDS, which goes to TXD of your serial adaptor, TP49 is RTS, and the black pin that I’ve stuck on the side of the reset switch is GND, because the outside of the reset switch is grounded, so you can just solder there, or anywhere else you can find ground.

Once the wires are outside of the Cybiko, you can then attach them to an RS232 DB-9 male connector. I suggest picking up a DB-9 break out board, like this one. Attach each of the 4 wires to the 4 points on the DB-9 connector matching TX, RX, RTS, and GND.

If in the very likely case your computer does not have a serial port, you can pick up a DB-9 Male Serial to USB adaptor like this one.

With all this, you’re ready to connect.

If you have a Windows XP computer to hand

A safety note regarding Windows XP: I strongly encourage you to not connect it to the internet if you can help it. It is NOT a secure operating system and should not have unfettered internet access. Copy files you need to and from it via USB sticks or similar.

Plug in your Cybiko via your serial adaptor or similar. If you are using the official Cybiko cable and are connecting to a real RS232 port on your computer, you can stop here. If you are using a USB to serial device, you might need to install its drivers: Open device manager (Start > Run > devmgmt.msc > OK), and verify that you can see your serial device (It’ll be under Ports (COM & LPT) if it is installed correctly and no yellow triangle will appear on it). If you find it under Unknown devices or there is a yellow triangle, you will need to install the drivers for your serial adaptor. Follow your vendors instructions for how to do that.

If you do not have a Windows XP computer to hand

One solution to not having a computer of this vintage is to virtualise one. This isn’t too difficult.

1. Plug in your Cybiko via your serial adaptor or similar.

2. Install Virtualbox. It can be gotten from your favourite Linux package manager, or see here for download links for less good operating systems.

3. Acquire installation media for Windows XP. I can’t help you with this one.

4. From within Virtualbox, select the “New” option.

5. Give your virtual machine a descriptive name, and ensure that Type is set to “Windows XP” and Version is set to “Windows XP (32 bit)”. The 32 bit option is important. Ask me how I know.

6. Select whichever RAM size you would like, 1GB or greater recommended.

7. Create the virtual disk with whichever size you like.

8. After the wizard has finished, right click it and select settings. Select USB on the left, and then click the USB Plus (+) icon on the right. A list should pop up listing all your USB devices. One of these should be your serial adaptor. Select it.

Note: If you do not see any USB devices listed, you may be running Linux and therefore may need to run sudo adduser $USER vboxusers in a terminal in order to add your user to the Virtualbox users group. If you’re not on Linux, dunno lol.

9. Select “OK”. Double click your virtual machine.

10. From the popup that appears, select your Windows XP installation media and follow through with the Windows XP installation procedure.

11. Once you are at the desktop, you’ll want to install the Virtualbox Guest Additions. You do this by selecting “Devices” at the top of the Virtualbox window, and selecting “Insert Guest Additions CD”. Then open “My Computer” from within the virtual machine, open the CD that is now there, and select “VBoxWindowsAdditions.exe”.

12. Follow the installation prompts, and reboot when prompted.

13. Open device manager (Start > Run > devmgmt.msc > OK), and verify that you can see your serial device (It’ll be under Ports (COM & LPT) if it is installed correctly and no yellow triangle will appear on it. If you find it under Unknown devices or there is a yellow triangle, you will need to install the drivers for your serial adaptor. Follow your vendors instructions for how to do that.

Cyberload setup and initial connection

Cyberload is a piece of software released with the Classic. It was superseded by EZ Loader, however I actually prefer Cyberload for Classics. Its purpose was to download the game/app catalogue from Cybiko.com (long gone), and to send games to the Cybiko.

1. Get a copy of the UK Cyberload from here, or the US version of Cyberload from here.

2. After installation, open Cyberload. If it doesn’t pick up on your Cybiko being plugged in, try powering the Cybiko entirely off (hold the escape key for around 10 seconds) and switching it back on while connected to the computer. You’ll know you’re connected once the Cybiko displays “Connected to PC” and the software shows your Cybikos ID on the right hand side. If you cannot get past this step, ensure your connections to your RS232 adaptor are correct!

3. You can now peruse the ‘Help’ section within Cyberload to learn about the various features of Cyberload.

Uploading software

To upload software to the Cybiko that you have found (for example, from here), follow the below instructions. I should also note I built the Cybiko Game Archive which lists every game that was ever released for the Cybiko, along with a description.

1. Counterintuitively, we need to move a file from your Cybiko to your PC, so that Cyberload creates directories on your local machine. To do this, select a file from the right hand side of Cyberload, and then press <= Move.

2. Check C:\Program Files\Cybiko\CyberLoad\Local Files and you should now see a folder called From Cybiko Computer XXXXXXXX where XXXXXXXX is your Cybiko ID. Inside there will be a folder called Device Flash. Copy your desired software into there.

3. Close and reopen Cyberload. The software should now appear on the left, and you can move it using the move button.

4. Close Cyberload, and disconnect your Cybiko.

Updating your Cybiko to the latest CYOS

The latest version of CYOS for the Classic is 1.3 System Pack version 57/58. 58 is for the UK, 57 is for the US (this determines which RF profile your Cybiko uses, since apparently there are different laws between the US and the UK). You can see which version your Cybiko Classic is running right now by pressing the ? button at the desktop.

If you need to update it or simply would like to factory reset it, the process is relatively simple.

1. Connect your Cybiko to a power source and also your computer.

2. Grab the Cybiko-Autoupdate-UK.exe file here or the US version here

3. Run the update and wait patiently for the update to complete.

Dealing with the Dark LCD problem

There is a problem I have seen with numerous Classic Cybiko units where the LCD becomes almost unreadable with extremely poor contrast. Not entirely black, but only viewable from extreme viewing angles. It seems to me this problem is afflicting many Cybikos. I believe is due to their age, but have no further diagnostic insight sadly.

One solution I came across was to use a piece of Cybiko software I recently came across called LCD Setup 2,

1. Grab LCD Setup 2, which can be downloaded here. Push that to your Cybiko using your preferred method.

2. Run the software on the Cybiko and be careful, the wrong options can result in a white screen, requiring a full software reset to fix.

3. There are 5 values that are configurable here. I suggest firstly moving the cursor to the fifth value (the contrast), and increasing it to something like 11. The other 4 values control the ‘colours’ the cybiko has. For my particular unit, the following values work well (31, 18, 14, 00, 11).

4. Press enter to persist, and then escape out of the app. Enjoy!

The problem

I recently became dissatisfied with my motherboard audio solution, so I investigated DACs and found the Fiio K3. The difference compared to the onboard motherboard audio was that the headphones could be driven louder (unsurprising) but the bass specifically seemed ’tighter' and more defined. I hate describing sound, but it sounded (ha!) as if the headphones were more “alive”. I’ll leave further discussion of how headphones sound to the audiophiles, but suffice to say I fell in love with this setup. What I did not love however was the cable clutter. The Fiio K3 connects to your computer via a USB-C cable, and your headphones plug into the DAC. This means you have the DAC/AMP sitting on your desk somewhere, and you must have a free USB port at all times for it, as well as the space to manage the cable that comes off the headphones (Most higher end headphones come with comically long cords, and unfortunately one downside to the Beyerdynamic DT880s (my headphones of choice) is their lack of removable cord).

I also use a ModMic USB. It is a fantastic microphone that sticks to the side of your nice headphones with a magnet, and plugs in via USB. This of course means you’ve now got two cables coming from your headphones, which of course you are going to want to cable manage together so they act as one cord, and on the other end you’ve got the headphone port which plugs into the DAC, and the ModMic USB port which plugs into a USB port. This means you now have two USB ports you have to have free on your computer at all times, which is not ideal. Ideally, I’d plug both the mic, and the headphones into my cases front IO ports.

What if instead of this mess, I just got a PCI-E sound card? That way, I’d just plug the headphones into the back of my PC like a normal person and be done with it. This solution is almost certainly the better solution for those who have a spare PCI-E slot, but I do not. My desktop computer is an ITX build (really small!), and has exactly one PCI-e slot which is used for a graphics card.

My awful solution

Before I continue to explain what I did, I think you probably shouldn’t do this. It’s a dumb idea, and is potentially dangerous for your hardware. If you’re not completely confident with what you’re doing, a number of things could go wrong. If you connect your DAC to your motherboard incorrectly, you could damage your motherboard or your DAC, or both. I obviously take no responsibility. If you do end up doing something like this, let me know and tweet at me!

I wanted to make use of my cases front headphone port, as well as an internal USB header. They weren’t seeing any use anyway, so I set about creating two cables which would allow me to mount my DAC inside my computer, so I could forget about it and just plug one headphone port and one USB port into the front panel of my computer.

Firstly, I needed to connect the DAC to a USB header. Of course, internally most motherboards do not have standard USB ports, but what they do have is USB headers which are usually used to connect to your cases USB ports. In fact, my AMD motherboard has what it calls a “AMD USB LED” header, which as far as I can tell is designed so those with certain types of AMD coolers with LED lighting can plug their cooler into that USB header, allowing software control of the lighting.

I do not personally bother with lighting inside my computer, so this port was free. I hacked together a USB-C to USB2 header connector as shown in the diagram below, and tested it worked by connecting my phone to it. Surprise, it’s just a USB header. This meant I could now connect my DAC to a USB header inside the case, freeing up a USB port.

All you need to determine on any USB header to know which way to connect the wires is which is the 5v and which is the ground pin. They will both be on the edges of the header, so you can easily check this with a multimeter.

Next, I needed some way of connecting the “HD audio” connector from my case to my DAC. It turns out HD audio is just normal headphone audio (one wire for left, one for right, one for ground), plus a bunch of other pins unnecessary for my purposes (the “sense” pins which sense when a device is plugged into the headphone port, as well as a second set of identical pins but for the microphone port, both unnecessary for our purposes here). So I soldered together a cable which has a male AUX port on one end that is plugged into the DAC, and a few male Dupont pins on the other end (the kind you’d plug into a breadboard) that we’ll jam into the HD audio connector.

I spliced an aux cable, and used a multimeter and a diagram of which part of the aux jack was for what to determine which wire was which. There doesn’t seem to be a standard colour scheme as there is for USB so this step definitely requires a multimeter. This one is probably safe to try to determine via trial and error, however. YMMV.

I then used some double sided tape to mount the DAC inside my case, and plugged my headphones and USB microphone into the front panel ports. This setup is significantly cleaner, and means I’ve effectively added a high end sound card to a build that definitely couldn’t fit a proper PCI-E sound-card.

I’d love to know if anyone else pulls off this silly trick, so please tweet me or send me an email if you do!

I recently started a diet. One thing I decided this time around was to be heavily data driven in my weight loss, and therefore I decided to get some ‘smart’ weighing scales. I got the Eufy P1 scales which seemed to be reasonably well reviewed, and it turns out that Eufy is another name for Anker, another Chinese brand I’ve had success with in the past. I was a little irritated though when I figured out there was absolutely no way to actually get your data out of the app they provide. I wanted to get the data out of the app since the graphs that the app provides were woefully inadequate, and to me were useless. This set me on the path of building a pretty fun Goldberg-esque system to extract the data.

What it does

The whole process is started when I weigh myself in the morning. After I conclude weighing myself, I take a screenshot of the screen which shows my stats from that weighing session.

Next, I have Nextcloud set up locally on a home server, and my phone is configured to automatically upload all screenshots to my Nextcloud instance. My home server then will occasionally (via a cron job) upload the screenshots to my server that hosts kn100.me.

Once the screenshot has found its way to my server, a Go service I wrote notices the new file, and kicks into action. It immediately calls out to a Python script which does some pretty crazy things which I will describe in more detail later, and then returns a JSON representation of the data in the screenshot. The Go program then takes that JSON from stdout (sorry), and writes it to a Sqlite DB, and eventually the frontend will make a call out to an endpoint which returns more different JSON for the graphs to display.

What about that Python script?

This is fun. My first attempt was completely written in Go. As you can see from the screenshot above, there are metrics spread evenly throughout the image. What I decided to do was to attempt to perform Optical Character Recognition on each metric. I found Tesseract to be a pretty good candidate for OCR, but running the entire screenshot through Tesseract was not an option, since the results of that, were…not fantastic. Tesseract seems mostly optimised for parsing blocks of text like pages from a book or receipts. Not so good for parsing freeform screenshots with text in various colours.

Instead, I firstly wrote some code that given some measurements from a screenshot, calculates where each metric is, and then crops out a square around each.

I then ran these through Tesseract, and got merely OK results. It seemed that Tesseract wasn’t getting on with the fact the unit text was smaller than the metric itself, and was occasionally confusing the last digit and the first part of the unit text to be part of the same character.

I thought about it for a while, and decided I needed some way of trimming off that unit from the end. All I cared about were the numbers, given I knew what each number at each position meant. I firstly tried a Tesseract whitelist, which would tell Tesseract only to look for certain characters. I set the whitelist to only allow digits and periods. This helped a bit, but occasionally the unit text would then be picked up as numbers, messing everything up. I really didn’t want to get into writing my own bounding box logic however, so I thought about cheap hacks to get rid of the smaller text to the right.

What I was doing in the beginning was silly. What I’d do is enormously gaussian blur the image, increase the contrast, and then sharpen it. Because the smaller details in the image (the unit text) would be less dark on the blurred image, when I re-sharpened it, those details would mostly be gone. By fine tuning how much I blurred the image along with how much brighter I made it, I was able to achieve surprisingly acceptable results. This improved OCR accuracy significantly. This approach had its own problems though, and would either remove the periods, or the remnants of what was left of the unit text would be interpreted as characters. I did try fudging it by attempting to parse the output to add in the dot where it would likely be, and to trim trailing dots, but still the accuracy wasn’t up to the standard I expected. Below is a visualisation of the blur, increase contrast, sharpen approach I was taking

For illustrative purposes I tried to reproduce the effect in Gimp, but wasn’t as successful as I was in code. The above above was as good as I got. You can see all four stages of the process here though. Firstly we blur the crap out of the metric. Then, we jack the contrast way up, and then we sharpen. I was actually able to get much better results programatically by fine tuning the numbers. The results I got from the blur, increase contrast, and sharpen approach were far better than the above visualisation lets on.

I was at my wits end at this point, and in my frustration joined #tesseract on Libera, an IRC network. I immediately asked my question, and had an absolutely hilarious conversation with a very kind soul there about my problem. Why hilarious you ask? Because #tesseract has absolutely nothing to do with the OCR software, and instead is a community around an open source FPS game. Amusingly, one of the members there still provided me with an absolutely genius solution which almost completely solved my problem.

kn100: I’ve got images which look like this https://nextcloud.kn100.me/s/8enRb4d9KQ2z2oE and all I want to do is detect the numbers, not the ‘kcal’ portion. With a whitelist set to just numbers, it occasionally detects the ‘kcal’ portion as numbers - I’m guessing Tesseract is choking on the fact there is a size difference in the font. Any suggestions on how I can configure Tesseract to better handle this? It’s not really an option to crop off the kcal before feeding it to tesseract - without something smart since it is unknown where exactly the numbers and the text appear in the image

@graphitemaster: has changed the topic to: Tesseract | http://tesseract.gg | Open-Source FPS Game | First Edition (May 11, 2014) released! | not tesseract-ocr | Calinou: we do OCR using pulse rifles here

@graphitemaster: kn100, Afraid you got the wrong channel. The topic wasn’t up to date since the channel moved from Freenode to Liberachat. This is a video game, not the OCR software. Though to answer your question, you could preprocess the input with opencv in Python first to remove the kcal before passing to Tesseract.

@graphitemaster: I’ve done some OCR stuff before to scrape codes from still frames and had similar problems. I found it was easier to just preprocess the input first with some CV to normalize and make Tesseract’s job easier. Something like that, if you use cv2, transform to greyscale, then use cv2.findContours, you’ll get actual bounding boxes around each character, you can very quickly tell that some of the characters will have a smaller area (maybe height is a better criteron) and you can just throw them out.

kn100: graphitemaster colour me surprised that I got help from you at all then :D Thank you so much, and sorry for misunderstanding :P

@graphitemaster: Stick around, might be some code in a minute XD

kn100: that is rather hilarious though. I might have to give Tesseract a go!

@graphitemaster: kn100, https://pastebin.com/raw/i6Zj9ebZ - t.jpg is your input

Their suggested approach was to use OpenCV. With OpenCV, I could do something a little more clever than just blurring and resharpening the image. Instead, I could use OpenCV to to find contours in the image (the edges of areas of high contrast), and measure the area of the bounding box around each contour. If a given contours bounding box wasn’t more than half of the area of the largest we’ve seen, we can throw away the pixels at that location. What was left were just the characters of the metric, which could then be stitched together again. This left me with just the numbers.

This was absolutely wonderful, and I immediately tried to port this approach to Go. Problem is, (and I mean no disrespect to the OpenCV devs!), OpenCV is a large beast. The lib that interfaces with it for Go was one called GoCV, and I just couldn’t get it to work. It was definitely something I was doing wrong but I was more interested in solving the problem at hand, so I quickly decided to do all the data processing in a Python script, which I would call out to from Go. It was a little messy, but whatever, it would work!

Onto Python, it worked great! There’s a package called opencv-python which brings along some precompiled binaries, which worked for my use case. Upon testing the approach that was kindly suggested to me in #tesseract, I realised it only partially solved my problem, as it omitted periods in metrics. This was easily rectifiable however, by adding a lower bound to the area of characters that get thrown away. Again, by fine tuning, I was able to reliably extract just the number portion of the screenshot.

Next up was the date parsing. This would be straight forward, but the EufyLife app has a fairly insane date format that seems to arbitrarily change. Sometimes days would have one digit, sometimes they’d have two. Sometimes hours would have one digit, sometimes they’d have two. This led to me writing some fairly horrific date parsing logic that I’m not proud of to get the date into a regular format to be parsed.

Past this point, the OCR results were almost perfect, but still not 100%. Often, the OCR would confuse the digit ‘5’ for ‘9’. What seemed to help a lot here was ’eroding’ the text to reduce how ‘bold’ it was, as well as introducing more space between each character. After this, the output from Tesseract has been entirely accurate for every metric I’ve checked.

Deploying it somewhere

One thing I was not prepared for is what a pain in the proverbials deploying Python code is. I’ve gotten so used to compiling a Go binary and having it just work, that I wasn’t prepared for deploying Python. Suffice to say, I ended up containerising it, but for reasons I am currently not completely aware of, the Docker container has to have a bunch of X11 related dependencies installed for OpenCV to work. I know that I’ve got some learning to do here. Python is not a language I’ve ever really used before, but writing Python was fairly enjoyable. I can see why Data people seem to really like it. I might end up rewriting the entire lot in Python, or I might end up trying harder to get OpenCV to work in Go, but for now this solution means I can weigh myself in the morning, take a screenshot, and about 20 seconds later it’s in a dashboard with graphs that aren’t entirely useless. Result!

A message to Eufy

Please, please provide some easy way to access this data. I don’t care if it gets exported as a CSV or if I have to query some API, just please give your users their data on request. Your in app graphs are absolutely useless, and even if they weren’t, why not just give this data up? For people like me who are a bit insane, we like data. we collect stupid amounts about ourselves. Your product becomes infinitely more valuable when you allow the user to extend it as they see fit. I know you’ve got the data, you’re logging it to some form of API somewhere, and even exporting it to Google Fit/Fitbit/Apple Health. Why not just allow me to query that API? The data is mine, give it to me!

Firstly, I’ll recount my contact with Terramaster. On Feb 21st 2021 I contacted Terramaster informing them of the issue. I contacted them again soon after again letting them know that changing the web server port does not help. I then contacted them a third time letting them know that this could be fixed fairly easily with a simple change to /etc/upnp.json as I mentioned in my previous post.

Terramaster replied on Feb 22nd saying they’d get someone to look into it, but also stated that this was intentional behavior:

Hello Kevin,

Thank you for contacting TerraMaster.

As for the mentioned security issue, it has been transferred to the R&D department, they will test here and fix it if it’s confirmed.

For port 8181 and 5443, it’s opened by default for it’s the default port of TOS webpage access.

I replied:

Hi,

Thanks for the confirmation that somebody will look into this.

Opening these ports via UPNP is not necessary for local access, and in fact makes the port accessible worldwide, externally.

I can also access my NAS via http://some-public-ip:8181 - on the public internet.

I hope that makes things clearer. Please me me know if you decide to change this behavior and if you do plan to fix this, please let me know what kind of timeline you’re working to. I’d like to publish a blog post about this (specifically around the danger of UPNP) once you’re happy everything is secure.

I then made the blog post 6 weeks later, hoping it might motivate them into fixing it. It sparked a lot of interesting discussion on Hacker News and Reddit /r/datahoarder. It blew up way bigger than I expected, and is by far the best performing blog post I’ve written. Pretty sad considering I threw it together in under an hour and didn’t even spell check it!

The issue specifically is that while the NAS claims its web interface is only locally accessible, it is actually accessible from anywhere in the world to anyone who knows the public IP of the network it is sitting on. This is because most consumer grade routers like the one I use have a feature called uPnP, which allows applications on your internal network to punch open ports that they need to operate. This feature is supposed to be useful for things like VOIP applications and gaming. The NAS for some reason decided to punch itself some ports open so that it could be internet accessible.

An interesting detail is that it punched more ports than you’d expect. It punched the default port that the web interface runs on (8181), a port for an SSL configured web server (5443), and then it also punches ports for 9091, as well as port 8800, which I’m not sure what the purpose is. My guess is that these ports were being used by a Terramaster developer, and they didn’t bother to remove the config to punch them before releasing the software.

Some suggested I just disable uPnP on my router, but I personally believe this kind of misses the point. I am a software engineer who regularly dabbles in infrastructure, and therefore could reasonably be expected to figure out how to disable uPnP. Thinking about the average consumer, who wouldn’t have any idea what port forwarding or uPnP is, I’m not so sure they could. Disabling uPnP definitely closes this particular security hole, but the vast majority of consumer routers are going to have this feature on, and therefore the vast majority of these NAS products in the wild are likely exposing themselves unwittingly. I am hoping that Terramaster will fix this issue in their product, and therefore unwitting users who aren’t aware of what uPnP is suddenly find themselves more secure, assuming they update!

Others suggest buying a new router which would support having two separate LAN networks so that this gear could be put on a LAN that is denied direct access to the internet/not have uPnP. This again I don’t see regular consumers doing en masse. I could do it, and very well may in the future, but I don’t really feel the need to upgrade my networking gear currently. It works, and I am happy with it.

This is why I went spelunking into the device itself. Thankfully Terramaster provides root on these boxes, which is nice of them. This led me to discover the fix I did.

Either way, one detail I got the most comments about was whether disabling uPnP was really a hit to convenience. Somebody emailed me questioning this specifically, as well as pointing out that GRC has a check to make sure your router isn’t allowing services on the internet to punch open ports on your router to hosts behind it. I had no idea this was a problem, and facepalmed pretty hard that this ever was, but nevertheless I replied:

If I’m being totally honest with you, this was more of a ‘feeling’ than it was an analytical decision. I should have known better than to assume I’d get away with that in a blog post like this, but I wasn’t quite expecting it to blow up the way it did!

I am reasonably confident in my router (author sidenote: lol), it’s actually not the trashiest ISP provisioned router I’ve ever had and the GRC uPnP exposure check is not flagging anything up.

I am strongly considering writing a blog post in future addressing this though - essentially my plan is to disable uPnP without making any other changes and seeing what breaks, if anything. It is completely possible I am very wrong that this is necessary these days. I am already port forwarding certain stuff I need externally accessible, but I’d be curious if the various consumer level stuff I’ve got floating around manages to keep working.

Thanks for contacting me!

So here I am, money where my mouth is. I’ve disabled uPnP and NAT-PMP on my router entirely.

Turning uPnP off led to some rather amusing error messages in my routers debug log

I’ll come back to this in a months time and see whether there has really been a hit to ‘convenience’ or whether I was truly wrong on this. I’m always happy to learn!

Addendum: The most controversial detail in this blog post so far has been my claim that disabling uPnP is a significant hit to convenience. I wanted to be fully transparent and say that I do not have data to back this claim up, and this is more of a feeling than it is an analytical decision. This criticism somewhat misses the point however, since the vast majority of consumer routes I’ve encountered personally have uPnP enabled by default, and the consumers purchasing this NAS might have no idea what uPnP is or how to disable it. I plan a blog post in future where I will disable uPnP without making any other changes, to see if anything breaks at all, because it is completely possible my opinion on this is wrong.

I have released another blog post related to this one, which provides more detail: Ok, Ok, I’m Turning uPnP off.

Terramaster appear to have released a fix for this, although I have not tested it yet. Read more

I recently bought a Terramaster F2-210. It’s a reasonably nice NAS that does what I ask of it. I however discovered something which unsettled me. As I’ve discussed in previous articles uPnP is a convenience that can be particularly dangerous. These NAS products are generally administrated using a web interface and The Terramaster TOS software is no different. The software requests you visit the hostname of the device on your network port 8181 in order to access the NAS interface, and the interface openly claims the NAS is not publicly accessible.

A few days after installing the NAS, I discovered I could access the NAS using my public IP, even though I hadn’t port forwarded anything! Upon inspecting my routers port forwarding rules, I identified that the NAS was punching 4 ports using uPnP. It was punching 8181 as we just discovered, but also 5443 which is for SSL access should you have configured it, and inexplicably port 9091, which normally is for Portainer, a container management tool for Docker, as well as 8800 - I’m not sure what this port is. It seems that potentially some of these rules were left in from the development process. I trust this NAS to be reliable hardware, however I am dubious of trusting its web interface to the open internet. Generally good practice to expose as little as possible to the public internet anyway!

Unfortunately, disabling uPnP these days is too much of a hit to convenience, so I looked for other solutions. My router is an ISP provisioned one so the feature-set there is somewhat limited, so I wanted to prevent the NAS from opening these ports rather than firewalling them off at the router.

I dug through the NAS interface and was not able to discover a way to disable this behaviour, since I didn’t really want my NAS administration interface publicly accessible. What I did discover was that the default web server port was at least configurable, so I changed it, and checked the uPnP port mappings again. Somewhat surprisingly, it punched holes for these new ports, but didn’t clear the existing ports! The NAS uPnP rules it punches are 5443, 8181, 9091, 8800, 54633, and 54632".

This annoyed me, so I contacted Terramaster about this 6 weeks ago, hoping they’d have a suggestion or something as a fix, but this wasn’t supplied, and therefore I went digging myself.

Upon SSHing into the NAS and having a dig around the file system, I discovered a file that could be modified. /etc/upnp.json seems to contain a list of port forwarding rules. Thank you to Terramaster for providing root access to these at least. Simply change bEnable to 0 for whatever ports you don’t want exposed, reboot the NAS, and check the port forwarding rules.

   "triestimes": 3,
   "mapList": [
       {
           "desc": "ftp",
           "nExternalPort": 6221,
           "nInternalPort": 21,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "ftp_data",
           "nExternalPort": 2000,
           "nInternalPort": 20,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "sshd",
           "nExternalPort": 22,
           "nInternalPort": 22,
           "sProtocol": "TCP",
           "bEnable": 1
       },
       {
           "desc": "telnetd",
           "nExternalPort": 23,
           "nInternalPort": 23,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "http_ssl",
           "nExternalPort": 54633,
           "nInternalPort": 54633,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "http",
           "nExternalPort": 54632,
           "nInternalPort": 54632,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "pt",
           "nExternalPort": 9091,
           "nInternalPort": 9091,
           "sProtocol": "TCP",
           "bEnable": 0
       },
       {
           "desc": "http_pri",
           "nExternalPort": 8800,
           "nInternalPort": 8800,
           "sProtocol": "TCP",
           "bEnable": 0
       }
   ]

This almost completely resolves the problem, however it seems the ports 8181 and 5443 remain punched, but result in a 404, since we moved the web server earlier. This is better than before, but still not perfect. I await further instructions from Terramaster/a software update, and will update this blog post should I get this.

One particularly scary change that happened recently was Google Photos deciding to change their ‘unlimited’ storage plan to not be unlimited anymore. This was obviously going to happen at some point, but frankly I feel like companies need to stop pretending like they can offer unlimited use of something unless they plan to honour that for the lifetime of the service. Either way, I’ve always felt pretty uncomfortable with having my photos being hosted and managed by someone else, so I started to explore alternatives.

Storing the photos

The first problem I had to solve is storing the photos. All in all, after doing a Google Takeout (to get the photos out of Google Photos) - it turns out I have about 50GB of media. A lot of it is crap for sure. I didn’t want to store these files on my main computer, since I regularly wipe and reinstall the operating system on my main computer, and therefore having to shift 50GB of data around to avoid wiping it would prove to be a pain. I also don’t back the entirety of my computer up, given I feel no need to. I back up the most important stuff, and that is it.

The solution I came upon was a NAS. I ended up purchasing the Terramaster F2-210 NAS. It’s pretty much one of the cheapest NAS units one can get here in the UK, costing me around 100 GBP. I also picked up 2 4TB Toshiba N300 drives, being the cheapest CMR drives at that size range you could get. You absolutely should make sure you don’t end up with SMR drives, they’re inappropriate for this sort of use case and your NAS performance will suffer. To understand why, read this excellent blog post from ServeTheHome. If you need larger CMR drives, maybe consider the large 8GB Toshiba N300 drive or if money isn’t too much of an issue they go all the way to 14TB. The 4TB drives I got were around 100 GBP each, bringing the total cost for this setup to around 300GBP. Not cheap, but given I had no decent redundant storage, I felt it a price worth paying.

This particular NAS is an ARM based unit, with a ARM V8 64-bit quad-core processor, with a frequency of up to 1.4 GHz, 1GB of RAM, and comes running Terramaster OS, which seems to just be a pretty light distribution of Linux. I’d much prefer running my own OS on such a unit, but part of the reason I purchased a NAS rather than building my own was I wanted something that “Just worked” out of the box. It’s got two hard drive bays, and supports both EXT4 and BTRFS for the underlying file system.

If you’re going deeper down this rabbit hole and want something a bit more extreme, you could get one of the Intel 5 bay NAS units, Like the Terramaster F5-220 unit which sports an Intel processor which will be significantly faster, and some are even experimenting with running FreeNAS on this unit!. These Intel based NASes may even come with the possibility of RAM upgrades, as well as some interesting hardware hackability possibilities.

It was pretty painless to set up, just pop the drives in, select whether you want RAID 0 (where all the storage is available to you) or RAID 1 (where the data is mirrored between both drives so if a drive fails the data is safe on the other drive), create some shares, and configure your users access to said shares. It also supports encryption of particular shares, although please note an encrypted share will have somewhat lower performance than an unencrypted share and also you cannot mount unencrypted shares via NFS. You’ve gotta stick with SMB for them. Performance of the NAS was surprisingly good, easily maxing out the Gigabit ethernet connection it has.

I then migrated all my photos over to the NAS, and was happy, as now they were somewhat more safely stored than they were prior.

What about a house fire? What if the NAS gets stolen?

One problem with storing your data locally is that if your NAS gets destroyed or goes missing, your data goes with it. This is why off site backup is important. I wanted to back my data up but for it to have the following properties:

• Important personal data is encrypted at rest on the NAS
• All data is encrypted at rest on the cloud backup solution
• The cloud backup solution is as cheap as possible
• The cloud backup solution doesn’t do the whole Amazon Glacier thing of charging you a fortune to retrieve your data
• The backing up of data happens automatically

For the cloud backup solution, I settled on Backblaze B2. The cost for backing your data up there is supremely cheap in comparison to Amazon S3, and Backblaze to me anyway seem like cool people. The cost for backing up 4 whole terabytes of information there with no retrieval over the course of a year would be around 240$.

I split the data on my nas into two categories: Stuff I’d be okay to lose in a house fire, and stuff I would not be okay with losing in a house fire. The stuff I’m worried about gets backed up to Backblaze, which is around 65GB of data total. The cost for backing this up over a year is $3.90 - a cost I can definitely stomach. You can calculate the cost yourself here.

As for getting the data to Backblaze, I picked Duplicati. Duplicati is a great tool that provides support for many backends (places you back stuff up) along with many options for how you’d like the backed up data to be managed, such as versioning of files and the such. It encrypts the data before it sends it to your chosen backend, and can run on a schedule, so it meets all of my requirements set out above.

I initially tried to run Duplicati on a Raspberry Pi, which turned out to be a hellish experience because of the dependencies it needs. I then tried running it on my main desktop computer, so it would read the files from the NAS and back them up to Backblaze, but it turns out Duplicati really doesn’t like reading from SMB shares. I tried instead mounting the NAS share via NFS, but as mentioned previously you can’t as far as I can tell mount NFS shares that are encrypted on the NAS. It was then I remembered that this NAS incredibly supports Docker, and therefore I could run Duplicati directly on the NAS itself.

I actually accidentally deleted all my photos from my local computer recently, which is what spurred this whole project in the first place. Duplicati did a wonderful job of restoring them, which I was very grateful for given I had not tested restoring from a backup yet.

Terramaster kindly provides us with Root on the NAS, so setting up the LinuxServer Duplicati Docker image was a doddle, and the NAS quite happily runs this Docker image and backs my stuff up to Backblaze early every morning. I’m honestly quite impressed with this little unit. It’s built very well, the OS seems quite reliable, and it’s done everything I’ve asked of it well so far.

Photo access/search

One of my favourite things about Google Photos was the fact it did image classification on images you supplied it so you could search for words like “Cat” or “Giraffe” and it would show you a mostly accurate list of photos of those specific things. Facial recognition was also creepy, but pretty useful too. I wanted to have this locally, and the project that spurred this whole thing was a wonderful open source project called Photoprism.

Photoprism is a piece of software written in Golang which promises to be a photo organisation/access tool that can run on a Raspberry Pi. It does image classification using a Tensorflow model, and can manage large libraries of photos. Running it is as simple as pulling the Docker compose file and building it.

It is a truly amazing piece of software, but where to run it? I purchased a Raspberry Pi 4 8GB variant (since Photoprism says it wants at least 4GB of RAM, and I wanted to run other services on this Pi), and connected it up to a spare SSD (A Kingston A400 480GB drive) in a USB enclosure (A Orico 2.5 inch USB 3 Enclosure) I had lying around which it turns out Raspberry Pi can boot from now. This offers much improved performance over a MicroSD card and significantly better reliability. MicroSD cards are notorious for failing in a Pi, and various blog posts have talked about the improvements in performance and reliability that can be had by booting from an SSD.

One thing you should bear in mind is the thermals of your Pi. The Pi 4 runs pretty hot when working hard, and Photoprism will work it extremely hard. I purchased a passive aluminium case for my Pi which does an amazing job of keeping the Pi cool enough to continue operating at it’s stock clock speed of 1.5GHz, but I wanted to overclock my Pi and at a 2.1GHz overclock the case is overwhelmed. For this, I purchased a 50MM fan which I attached to the GPIO pins of the Pi and blu-tac’d it to the case, and now it does not exceed 60 degrees under extreme load and operates at 2.1GHz all day. You’ll also want a decent power supply.

After installing Raspbian and doing the standard hardening routine (disable root account, make a fresh user, disable ‘pi’ user, fail2ban, etc), I set Photoprism up on it, and pointed it at my NAS as the photo storage directory, but told it to store its cache locally on the SSD - since decent performance is probably a good thing for that.

It took around 3 days total for Photoprism to finish churning through all my photos and videos, where it generated almost 30GB of thumbnails, but after this process was complete I ended up with a lovely searchable indexed version of my photos, that while not quite as slick as Google Photos certainly meets my needs.

Adding more photos

Another problem is how do I back up new photos I’ve taken? I decided to go somewhat oldschool on this in the interest of keeping things simple, and set up Shotwell, a piece of free software which can import photos from your camera or phone, and organises them into a YYYY/MM/DD folder structure. It avoids reimporting duplicates, and seems somewhat okay with reading and writing to a SMB share. This means about once a week, I’ll dock my phone, import the photos using Shotwell, which will then put them on my NAS. At 4am the next day, those photos are backed up to Backblaze safely, and Photoprism will index them, making them available for search/retrieval.

This solution isn’t ideal, since Shotwell takes forever to be ready to import photos (assumedly because it’s reading the file list and checking nothing has changed from the SMB share), but it works well enough for me right now.

Deleting the photos from Google Photos

Once I’d done all this and was happy with the solution, I decided it was time to delete all the photos I had from Google Photos. Turns out, there is no button you can press to just delete all the photos you have, and instead I found a lovely script on Github called google-photos-delete-tool which when pasted in the dev tools of your browser, will delete your photos by automating the process of selecting them and moving them to trash. Be careful while doing this, since photos which were previously stored under the “Unlimited” photo storage plan will now take up space in your bin, and therefore you’ll need to empty your bin every now and then while the script is working. I had to regularly refresh the page and restart the script too, as it seemed to hang after deleting about 6 months of photos.

Conclusion

Thanks to a combination of a NAS, Raspberry Pi, Duplicati and Backblaze B2, I’ve got a photo management solution that I am fully in control of, that is housefire proof, and is encrypted at rest everywhere. I’m pretty happy about this, and given I use the Raspberry Pi and the NAS for other uses too (Plex, Home Assistant, etc) - it is fairly cost effective.

There exist many projects to de-cloud these products. One such project is called Tuya-convert. Tuya convert is a tool which emulates the update server these plugs connect to in order to deliver custom firmware to the plug that it can run. This project is amazing, since it gives you the option of declouding IOT gear without having to do any hardware modification at all. Unfortunately, it seems this project is dead in the water right now since Tuya is playing the typical cat and mouse game with the developers, and currently Tuya is winning.

I wanted to open my plug up next, in order to figure out what made it tick. Opening it was fairly difficult, given that it is held together with nothing but clips. After running a guitar pick around the seam a few times, I finally managed to pop the cover. What I found really surprised me. There was a board in there that looked suspiciously like an ESP based platform. Further searching led me to realise that the board in there that handles all the ‘smart’ of the plug is actually an implementation of the ESP8285 - which is a cheaper (but just as hackable) variant of the ESP8286, which is related to the ESP32.

Consulting the easily accessible datasheet for the TYWE-2s - we can quickly identify the serial pins, and solder wires to them. Then, we can connect it up to some Serial to USB adaptor and could flash whatever code we wanted to the ESP. I however wanted a nicer solution. I found Tasmota. Tasmota is another open source project that runs on these plugs that allows you to connect them up to HomeAssistant or similar. It works really well. Read on for the process:

1. Buy a WiFi plug.

2. Open it up, in order to figure out where the serial pins are. We can see on mine, there is a nice TYWE-2S module which unfortunately due the construction of this plug has awkward to access serial pins.

Above is a view of the pins we need access to. Unfortunately, the two options you have for getting access to them are to desolder the enormous blobs of solder holding the mains plug pins on, or to cut into the case. I went with cutting into the case. Ugly, but works.

3. Put it back together, and cut a hole where the pins you need are. Below is a view of the hole I cut. I cut it using a hacksaw and a hot knife. Uuuugly.

See here for a data sheet to help identify which pins are which.

4. Solder some female jumper wires to the pins we need access to, and connect them to the serial interface. You’ll want to make especially super sure that your interface is clever enough to support 3.3v logic level input. Otherwise you risk frying the board or just failing to flash the board repeately. Ask me how I know.

5. Grab Tasmotiser and follow the instructions. Make very sure your wifi details are correct, otherwise you’ll end up having to flash the plug a second time.

6. Once your Tasmotised plug is up, and you can control it from a web interface, it’s time to interface it with your HomeAssistant install. Your Home Assistant install must already have the MQTT integration. Ensure you enable discovery in your Home Assistant MQTT config.

7. Go back to the web server for your WiFi Plug. Configure the MQTT server to connect to the MQTT server your Home Assistant is connected to.

8. In this same interface, head to console, and type SetOption19 1. This causes the plug to emit an autodiscovery message which should mean Home Assistant picks up on your plug and you should now be able to control it in Home Assistant, sans cloud!

Bluetooth Audio is generally considered convenient, but not ‘audiophile’. Any self respecting audiophile is probably connecting their fancy headphones to some Digital to Analog Converter/Amp combo which might be connected to their computer using some fancy gold plated USB Cable. More power to them, I am not self respecting. I made a decision a few months ago to switch entirely to Bluetooth Audio as the convenience and tidiness of having a wireless headset just appealed to me so much.

I purchased the M-Pow H21s - which are a fairly budget noise cancelling pair of Bluetooth headphones. They appealed to me because they weren’t enormously expensive, and other M-Pow stuff I’d purchased in the past was of good enough quality for me to trust these would be too. I wasn’t wrong. The noise cancelling is reasonably good, and the sound profile, while not ’neutral’ - is totally fine for me to use while working.

One thing that really did annoy me about these headphones (and I’m not sure if the fault lies with Bluetooth Audio on Linux or with the headphones themselves) is that if I walked away from my desk and then returned, the audio quality would perceptibly drop, but never quite recover. It seemed to get ‘stuck’ at a lower quality, until the headset was rebooted. One day, I got tired of this, so decided to research what was going on.

Codecs in General

In the beginning, there was A2DP - or the Advanced Audio Distribution Profile. This is a Bluetooth profile - which describes a method by which audio can be transmitted between a sender and a receiver. The profile mandates that all Bluetooth devices support a codec called SBC - or Low Complexity Subband Codec. A codec defines exactly how the device that is sending the audio should compress it. SBC is what’s known as a Low Complexity codec. Low Complexity codecs have the design goal of being easy to encode for the sender and being easy to decode for the receiver. It’s a fairly old codec, being the precursor to MP2 - itself the precursor of MP3. When consumers started demanding better quality audio out of their Bluetooth hardware, various other codecs were added to hardware. Some hardware added MP3 encoding and decoding - which was a small improvement in two ways. Firstly, if the source material was already MP3, and the encoder was smart enough, the MP3 data could directly be sent to the receiver, meaning that no encoding step was necessary. Secondly, MP3 is generally considered to be a better codec in terms of the resulting audio quality, so even if there was an encoding step, the results were generally better. Apple added AAC support to their hardware - which itself is a far superior codec to MP3 lends much the same benefits. The freshest codec which seems to be available is from a company that Qualcomm gobbled up - aptX. This codec is interesting in the sense that it only offers the second advantage talked about above. Nobody has music in aptX format, which meant that all source material must be re-encoded on the device sending the audio.

A Better Codec, or a Higher Bitrate?

The first step to identifying if improvements can be made to your own Bluetooth Audio is to figure out which codecs your Bluetooth Headset supports. You can generally find this out by finding the manufacturers information. My particular headset, the h21 - is dumb as rocks, and only supports SBC. If your headset supports a better codec, especially if it supports a codec called LDAC - you should very much consider trying that before trying to do what we’ll be doing in this article, which is improving the quality available to us using the SBC codec. This article does a reasonably good job of covering all the codecs.

No matter what your headphones support, the next thing you’re going to want to do is install a custom PulseAudio module which both enables support for a bunch of more fancy codecs, and also allows you to configure the codecs. Go grab EHfive/pulseaudio-modules-bt and restart PulseAudio. You should already see in your systems sound settings you can now actually select between AAC, APTX, APTX HD, and LDAC - if your headset supports it. You can probably stop here if you’re able to switch to APTX or LDAC. If not, it’s time to configure SBC to be less sucky.

So, a quick review of the options the SBC encoder has that we likely care about. Firstly, the Stereo mode. Generally in audio the two stereo modes we have are.. well Stereo and Joint Stereo. Stereo transmits two distinct channels of audio in the same stream, but completely distinct from one another. Joint stereo makes the assumption that the left and right channels are probably similar enough that just encoding how the right channel differs from the left will result in a more efficient packing of data. Your headset is likely already using one of these two modes. SBC actually supports a third mode, called dual - which essentially sends two audio streams to your headset, completely independently of one another. This effectively doubles the bit-rate your headset can operate at, since you now will have two separate streams of audio being encoded, rather than just one. If your headset supports this dual mode, and it most likely does, you definitely want it. In fact, this unofficial feature has been added to a popular custom Android OS - called LineageOS, and they describe the benefit of this very well here.

The next option we likely care about is the bitpool. The bitpool effectively determines the bitrate the audio will be encoded at. The higher, the better. A calculator is available here but what is shows is that at the default highest available Bitpool value (53) in Joint Stereo mode - the maximum available bitrate is 328kbps. If we do nothing else, and just switch to Dual Channel mode - the bitrate predictably roughly doubles to 617.4kbps. Some headsets, like my H21s, actually seem to do fine at higher bitpool values. Through experimentation, I found that around 70 was as much as my headset could manage without hearing the audio equivalent of a buffer overflow. At a bitpool value of 70, in dual channel mode, the bitrate now gets to 804.8kbps!

If you are experimenting with a different headset, I suggest starting with a bitpool value of 53 and dual stereo enabled. You can then experiment with increasing the bitpool value, as described below.

To modify these values, you’ll need to edit /etc/pulse/default.pa. Firstly, find the lind that begins load-module module-bluetooth-discover. Modify it to add these flags, like so:

load-module module-bluetooth-discover a2dp_config="sbc_cmode=dual sbc_min_bp=53 sbc_max_bp=53 sbc_freq=44k"

This will lock your SBC encoder to work at a bitpool size of 53 - in dual stereo mode. Restart PulseAudio by doing pulseaudio -k - and reconnect your Bluetooth headset. Hopefully, you’ll perceive a large improvement in quality, as I did. You can now experiment by changing that value of 53 for a higher value. There’s a list of known compatibility available here.

The one downside of this approach of locking the encoder to work at a particular bitpool is that you lose graceful degredation of quality as the link quality reduces. You could experiment with having a lower value for the sbc_min_bp so that the encoder can reduce quality.

Let me know if this helps you!

Hacker News Discussion | Reddit Discussion

My requirements for a hub were very simple:

• it must be externally powered,
• it must be relatively aesthetically neutral - no gamer aesthetics or hideous glossy plastic,
• it must be USB 3,
• it must offer between 3 and 7 extra ports - any less and what’s the point, and more and the hub is unnecessarily big, and I treat my desk like Tokyo. Space is a at a premium,
• the cost must not exceed 35 GBP (45 US Dollars at the time of writing),
• and it must have a detachable cord that connects it to the computer, or offer a cord long enough to reach my computer under my desk.

While shopping, almost every single hub I came across had some stupid design that either prioritised it looking space age with blue LEDS galore, and a lot of them even had individual power switches for each USB port. Why anyone would want that is beyond me. Other hubs had tiny 15cm cables, since they’re designed to be used right beside a laptop.

I ended up settling on a very cleverly designed Orico 4 port Aluminium hub. I particularly liked this one because aesthetically speaking it was pretty neutral, and it even handles the problem of the USB hub floating off in the breeze by integrating a clamp so that you can clamp it to your desk or your monitor.

It arrived, and seemed to work perfectly. USB transfer speeds were as expected, the external power source being MicroUSB meant I could attach any fairly dumb USB power supply to it to supply extra power, the cable leading to the PC itself was detachable and relatively standard (USB A to USB A) and all was good with the world.

That was until I put my computer to sleep. The next morning, I pressed a key on my keyboard to trigger the system to wake up, and nothing happened. The keyboard is not attached to the hub, nor is it attached to a port on the motherboard that is on the same controller, so this was perplexing. I pressed the power button on the tower, but still nothing! Very strange. I detached the USB hub, and immediately my computer came to life and worked as normal.

Through further experimentation I determined this strange phenomenon only happened when an external power source was connected. I toyed with the idea of returning what was obviously a faulty product, but reasoned that I didn’t actually need it to be powered, and pretty much all the other hubs in the same price range were either aesthetically disgusting or lacked other features I liked about this one - so I kept it.

That is until today, when through some research, I found this isn’t just a problem with my USB hub, it’s a problem many have with powered hubs. Articles like this, or forum posts like this show that totally unrelated hubs cause similar issues for their owners. This got me thinking to past experiences of USB hubs I’ve had. In the past, a less financially stable me used to buy the cheapest possible thing every time. At one time, I was experimenting with a Raspberry Pi along with external hard drives. I discovered that a powered hub was necessary for these experiments because the Pi itself was incapable of providing much USB power. I bought the cheapest USB 2 powered hub I could at the time from eBay, and it mostly worked fine. To describe the setup a little, I had the hub itself connected to its power supply, and the ‘in’ usb port connected to one of the Pis USB ports. Imagine my surprise when the Pi switched on and booted up, without any power supply attached! I thought this was cool at the time, and chalked it up to ‘It’s not a bug, it’s a feature’, and forgot about it. Fast forward to today me, when I realised my experience back then was possibly related to the problems I was having with this new Orico USB hub today. After a little bit of thought, I realised that the hub was most likely ‘backfeeding’ power to the connected computer.

What is backfeeding?

To keep things simple, let’s stick to USB 2 for now. USB 2 consists of 4 conductors, known as VCC (this is the ‘positive’ 5 volt connection), GND (this is the ’negative’ power connection), D+, and D- (the data transmission pins, no true power flows over these pins). This means that a connected USB device can both draw power and transfer data. When you see 5 volts on a power adaptor or something, this is usually an approximation. For various reasons which revolve mostly around cost saving, most USB power adaptors will actually output slightly more than 5 volts, and then when a load is applied (say, a charging phone), the voltage drops down a bit. The better the quality of the power supply, closer to 5 volts it’ll start at generally and the smaller the drop when a load is applied.

This applies to your computer’s USB ports too - except for one big difference: the power supply inside your computer is probably significantly better designed than the USB plug sockets that are mass produced. In fact, if I measure the voltage coming from my USB ports on my computer, I get a value of around 5.03v. If I measure the voltage coming from my phone charger, I get 5.21v. This is a pretty big difference!

The next thing to know is how a powered hub actually works. Essentially, a powered hub takes an external power source, and passes that power source through to the USB devices connected to the hub in place of your computer. One problem that these hubs face however is what if the user wishes to use the hub in its ‘unpowered’ state - where they’ve connected the hub to their computer with no power supply. In order to implement this properly, the hub would probably need some mechanism to switch between the computer power and the external PSU if it is connected. This would however add cost to the manufacture of the hub, so instead of doing this, they just connect the 5v of your computer through to the 5v of the external power supply. This is ridiculously bad because if there is even the smallest difference between the voltage being supplied by your computer and the voltage being supplied by the external power source, power will flow towards the device with the lower voltage! This is ‘backfeeding’. As a simple analogy, imagine you took two rechargeable batteries and connected them together. What you’d find happens is that the batteries will eventually end up at exactly the same voltage, which will be slightly lower than what you’d expect if you did the maths, as a small amount of power would be lost as heat. If the power source is ‘infinite’ however - like our wall connected computer or our wall connected external hub, this power will just continue to flow. Where it goes or what happens with it is completely undefined. In the case of our Raspberry Pi earlier, the circuit was simple enough that it just led to the Pi being powered up. In the case of my computer, it screwed with the system so badly that hardware buttons like the power button literally stopped functioning altogether.

Why would they design it like this

This design seems stupid right? The problem is, it is the cheapest way of achieving the design goal of having a USB hub work with or without power. If we didn’t care about this particular feature, we could either just not have an external power source and have a purely unpowered hub, or we could not connect the 5v pin from the computer and therefore only supply power from the external PSU. Both of these solutions in effect are more expensive than just living with the backfeeding ‘feature’ since it’d mean the company offering the hub would need to deal with support requests from people wondering why their hub doesn’t work ‘unpowered’ when their other one does, or vice versa. Instead, we get backfeeding. I’m sure that some computers would not exhibit any real problems with this design, seeing as it’s been around since as long as powered USB hubs have been around as far as I can tell, and it’s likely some USB chipsets are designed with this in mind, but mine was not. Others computers, especially Macs, deal with this particular issue far more destructively, causing damage to the computer’s USB chipset when a backfeeding USB hub is connected.

Disclaimer

Before we continue, you should know I am not an electrical engineer, if that wasn’t clear from how vaguely I described things above. I am a software engineer who likes to dabble with electronics. This means my advice below does not come from someone who is qualified to give it. You should follow this advice at your own risk. I take no responsibility for damage you do to yourself, your USB hub, or your computer, or to anything else for that matter.

Fixing this BS for ourselves

1. Firstly, I validated this was the problem. I engaged in a clever little trick where I took an old USB cable - USB 2 or 3 is fine, cut it in half, exposed the either 2 (cable is power only) or 4 wires, and looked up USB wire colouring to realise that the red wire is usually the VCC wire, and Black/White is usually GND. I then connected the cut up USB lead to the input port on my hub (where my computer connects), and connected the external power supply. I measured 5 volts across these two wires using the multimeter, meaning this hub does backfeed.

2. My hub comes apart very easily. Disconnect all wires. Four screws removed and we immediately get access to the circuit board. USB 3 is a little more complicated than USB 2 - featuring 9 conductors rather than USB 2s 4.

3. I have no idea what the pinout of USB 3 is, nor how this specific connector orders things. I connected the cut up USB lead to the input port on my hub (where my computer connects). We can then attach our multimeter in continuity mode to the red wire, and used the other probe to probe about the USB port to figure out which of the pins is VCC. In my case, it was the 8th pin from the left.

4. I snipped this pin using my wifes beauty scissors. If you do this same thing, I recommend not telling your wife.

5. I reassembled the hub, and connected it only to power. I connected my multimeter in voltage measuring mode across both wires of our cut up USB lead. I measured zero volts. This is to validate the work we have just done. Our hub no longer backfeeds!

You may have seen my article on the air quality meter I built a while ago. Diyode, a print magazine in Australia kindly published an article discussing the process I took to create the thing! It’s a really cool feeling to be featured somewhere like this. The article is a great read, check it out over on Diyode: Sense of Smell.

This article was recently featured in Hackaday and Diyode!

I’m bored. I’m sure you are too. I’m personally very thankful that I can still work through this strange period - but there are still hours in the day to fill. I wanted a personal project, so I thought about remote working and the health impacts. Most offices here have air handling units, air conditioners, and fans to keep air circulating. Most residential flats like the one I rent do not. Given I now spend close to 24 hours a day sat in this flat, air quality seems important. I looked up the cost of air quality meters and they’re either fairly pricy or fairly useless. I wanted one that I could graph over time. I decided to DIY one! This wasn’t intended to be a super serious project, just a bit of fun and an opportunity to learn more about embedded systems.

In the end, I built a little unit I think looks pretty nice given my limited DIY skills that:

• Displays realtime air quality data on a LCD
• Reports realtime air quality data to a MQTT broker
• Allows realtime monitoring on any device that can do MQTT.

I also involved a Raspberry Pi in this party that:

• Persists data to an InfluxDB instance
• Hosts a Grafana instance
• Reboots occasionally and crashes if you look at it funny - just Raspberry Pi things.

The Parts

4MB Flash MIINI WEMOS D1 Lolin32 ESP32 Dev Board

I have worked with Arduino in the past - and have always found them to be a really fun tool to have in the arsenal. They’re cheap as hell and very useful in many electronics projects. I wanted something that was similar to work with in capability, but wanted more - namely I wanted to connect it to the internet! There are ways to connect an Arduino to the internet, but a far more seamless solution presented itself inside a gift my lovely fiance got for me a few months back - the Odroid Go! The Odroid Go is portable games console styled after the Gameboy and is capable of playing the games of many retro consoles. It also has a pretty prolific hacker community - since it is powered by a somewhat custom ESP32.

The ESP32 sounded incredible! two 240MHz cores + a third ’low power’ core to keep things ticking while it’s asleep, 520KB of SRAM, WiFi, Bluetooth, 34 GPIO Pins, 12 bit ADC, SPI and I2C support - it really sounded like the perfect hacker board. Even the ESP32 data-sheet does a good job of selling the applications of a solution like this, listing IoT Sensor Hub as its top application. Best of all, it’s unreasonably cheap! I picked up what I am pretty sure is a clone of another dev board manufactured by WEMOS from eBay for 7GBP delivered.

You can write code for the ESP32 in numerous ways - but the way that I was interested in was using the Arduino IDE - since I already had some experience with it in the past.

2004 I2C LCD Display

I wanted to have some sort of external display that would show the realtime values at all times. I considered going for a full colour display here, but decided that for simplicities sake I’d go for one of the old fashioned, text only ones that was probably intended to go into some industrial control hardware or a VCR. It also has the benefit of being fairly low power in comparison to a permanently backlit full colour TFT display.

The one I got has a lovely interface board already soldered on that allows you to talk to it using I2C - and also has a bright green backlight that my fiance has lovingly nicknamed the ‘Shrek’ lamp.

CJMCU-680 BME680 Module

This component it what sparked this whole project off. My phones assistant fairly regularly notifies me of Kickstarter or IndieGoGo projects that are fundraising - and this time the advertised product looked fairly interesting. It was the Metriful Sense board - an Indoor Environment Monitor board. It features a Light sensor, microphone, and the Bosch BME680. I was considering backing the Metriful Kickstarter and I’d encourage you to do so if this sounds good to you - however I decided I didn’t care about light and sound, didn’t want to wait months for it to shop, and only really cared about air quality, so I went to eBay and found that the BME680 is being sold on breakout boards for around 15GBP - shipped first class, so I got that instead. This component is actually the most expensive part of the build (I paid 16GBP) - but it is a pretty cool sensor.

The BME680 measures temperature, pressure, humidity, and gas resistance (how conductive the gas is) - and can offer you these values raw, or you can use a closed source Bosch library called BSEC which uses these 4 values and some magic sensor fusion tech to compute an Air Quality Index, as well as a guess at the Co2 and VOC concentrations in the room.

Enclosure

For an enclosure, I decided I wanted something fairly roomy - having been burned by trying to make miniature projects in the past. I wanted something that would fit the aesthetic of my flat, and something that was fairly easy to work with. I ended up settling on a plastic lunchbox which has a lovely wooden lid that press fits into the plastic from a shop called Flying Tiger. Really, any box that will fit the components and you can drill will work. I think it turned out pretty nice even with my absolutely terrible woodworking skills.

Battery Choices

I wanted to be able to transport this meter around the house and have it continuously measure for a period after it was unplugged. I wanted it to have reasonably good battery life. I also try to reuse things I have that would be ’e-waste’ otherwise and I settled on a 3000mAh 18650 battery that I had lying around that was in use for a high current application prior, but now had developed too high of an internal resistance to really push amps. Perfect for this project then, since the entire project will sip around 80mA in its completely unoptimised form, and this could easily be significantly reduced. In reality, the battery lasts around 30 hours. All I needed to purchase here was an 18650 holder.

Raspberry Pi or other server you control

The ESP32 is a very powerful dev board, but in order to persist large amounts of data and to make it accessible in something like Grafana we’re going to need something a bit more powerful! I had a Raspberry Pi 3b+ sitting in the ‘one day I’ll find a use for you’ box I have and decided this was a perfect use. It can be my MQTT broker. It can even perform double duty and host Grafana for me. I did consider hosting all of this on the server that hosts this very website (A Scaleway instance) - but didn’t want to pollute that environment too much if I ended up getting bored of this project.

I’d like to stress here something that I learned the hard way - a good quality power supply is an absolute requirement for any Raspberry Pi. Lower quality power supplies just aren’t beefy enough to really supply the Pi when things really get going and you’ll find yourself chasing ghosts. Get a decent power supply!

The Build

I2C Components

I’ll provide a quick and dirty primer on I2C: It’s a way of attaching peripherals to microprocessors over short distances. It’s pretty slow topping out at 5 Mbps - but is nice since it has extremely simple wiring (just 2 data wires!) and you can chain devices. Effectively, we just need to connect the SDA and SCL pins (along with power) between the components and the ESP32 and the hardware portion is finished. When we want to connect two devices, we just connect all 3 in a chain (connect all the SDAs together, connect all the SCLs together). The only other thing we realistically need to take into account is the address. Each I2C Peripheral will have an address that identifies it on the bus. These addresses are not unique. If you purchase two of the same peripheral, you’ll probably find they have the same address in their default configuration. This means that you cannot ’talk’ to each device independently if they’re on the same bus. There are a couple of solutions to this problem: the simplest being peripherals that feature an ‘address select’ jumper which will allow you to select a secondary address to use instead of the first, and other peripherals will be reprogrammable so you can change the address freely. YMMV.

So, to connect our I2C components up, just connect all the SDAs together, and all the SCLs together. Pretty simple!

Once we’ve done this, it’s time to power them.

Power considerations

In order to figure out how to power this all, I needed to know the power requirements of each component. In the case of the ESP32 - I am using a dev board which will have its own power draws on top of the ESP32 itself for things like the serial to USB chip as well as for the charging circuitry since it features a LiPo charging circuit, so the current requirement is from the ESP32 data sheet directly - since this weird clone board I got doesn’t seem to have a datasheet that exactly matches it. Same goes for the LCD and the CJMCU-680. This is the downside of buying strange boards from Aliexpress I guess! For this specific project, I experimented a little and did some voltage/current testing and got these values.

Let’s start with the CJMCU-680. It runs at 3.3v and needs 10mA, but only very briefly. As far as I can tell, gas resistivity sensors like the one found in the BME680 heat up a plate inside the component to ‘activate’ some material that reacts with various gasses - changing the resistance of this sensing material. The measurement of this resistance is how the sensor determines the gas resistivity. The initial heating phase which happens in less than 0.1s according to the datasheet is the highest current phase, at which point the current draw drops dramatically. I connected this component to the 3.3v and GND pins on the ESP32 and it seemed to work so that’s how I left it.

Next up is the ESP32 itself. The dev board I got helpfully provides two ways to power it. You can either just connect a USB power supply to the onboard MicroUSB port which powers the whole lot, or you can connect a Lipo cell to the onboard battery connector, at which point the onboard MicroUSB port doubles as a pretty bad battery charger. The charger on my board seems to charge the battery at around 400mA with all other components attached and working - which means it’s probably supplying 500mA max (respecting the USB spec for once!). I went with the Lipo option, but this has a small downside - the 5v rail that the board supplies then just doesn’t seem to supply enough current to power the LCD.

For the LCD, what you’d probably want to do is engineer your own power solution that bypasses the crappy one found on the dev board. You might get your own step up converter that will take the 3.7v from the Lipo and boost it to 5v with some current rating that makes sense (1 amp is more than sufficient). You can then also get your own LiPo charging circuitry that will charge the Lipo at whichever rate you like rather than the rather slow but safe 500mA charging rate. You could go belt and braces and add on a low dropout circuit which cuts the battery connection if its voltage drops below something like 3v - since anything below this and you’ll start to enter explodey battery territory. In fact, you can buy a single circuit board that will do all of this for you in a single integrated solution. Something like this 5v power bank module or this LiPo charge circuit would likely do. What I did and do not recommend you do is just connect the LCD directly to the LiPo. This means I have completely worked around the low dropout protection the ESP32 already has which is a Bad Thing(tm) - but for my given use case (almost always plugged in, rarely without power for more than 12 hours, very large battery) - I’m happy with it. In either case, make sure you connect the ground of the LCD to a ground on the ESP32 - you must always common (connect) grounds when using something like I2C.

Onto the battery! As previously discussed, I used an 18650 I had lying around along with an 18650 battery holder.

Enclosure

Quick note: Woodwork is NOT my strong suit, so please forgive the tragic method that I describe. I purchased a lunchbox that is a plastic box with a piece of wood that fits into the plastic box as a lid. I cut a small hole in the bottom so the BME680 sensor could breathe by heating a drill bit and slowly drilling through the plastic. I cut another three of these holes on the back, one sort of MicroUSB shaped, another for a toggle switch that I use to toggle the backlight of the LCD, and another push button switch that pulls io0 low so that my dev board goes into flash mode. The hole I cut in the wooden lid is rough as hell because I do not own a jigsaw nor did I want to buy one. Instead, what I did was I drilled holes all along the edges of the wood I wanted to cut out, and then stuck a hacksaw blade through these holes and cut using the hacksaw. It took forever and there was probably a better way available to me, but this method worked in the end. Several minutes of sanding to get the LCD to press fit into the hole, and it was finished.

I glued the BME680 in place using a adhesive rubber furniture foot thing because that’s what I had to hand. If this is a permanent installation, you may want to invest in some standoffs and screws so you can mount yours properly. I found on eBay a MicroUSB ’extender’ cable, that has a male MicroUSB port on one side and a female MicroUSB port on the other. I epoxied the female side to the microUSB shaped hole, and plugged the male side into the ESP32. I superglued the 18650 battery holder in place. The ESP32 is currently just floating about on the inside of the case since I still frequently access pins on it.

It goes without saying, internally this is a bit of a mess. I don’t really mind this as it is purely a hacker project for me, something to keep me sane during Covidpocalypse. It works well and externally I’ve grown to like how it looks.

ESP32 Software

Next up, we should write the code that runs the show on the ESP32. As previously discussed, I wanted to use the Arduino IDE initially since that papers over a lot of the horribleness of working with embedded systems such as this one. There is a fair bit of horribleness still - which I shall elaborate on. Due to incompatibilities between the BSEC Library and the Arduino IDE, I recommend installing Arduino IDE v1.8.10. This one works for me. Once you’ve got the Arduino IDE installed, you’re going to want to install the board tools for your board. This blog post covers installing the ESP32 dev tools. Ensure you are able to write software to the board. On my first attempt with this, the ESP32 point blank refused to accept code. I’d constantly get “ESP32: Timed out… Connecting…”. All the instructions everywhere told me to press the ‘Boot’ or ‘EN’ buttons, but my board only had one button labelled ‘reset’ and that didn’t help. What did help (and I can’t remember how or where I found to try this!) was while you are in that ‘Connecting…’ phase briefly connect the IO0 pin to ground. This seemed to kick it into ‘flash mode’, and it flashes correctly every time.

Here is a list of the libraries I used:

• BSEC Library by Bosch Sensortec (v1.5.1474)
• PubSubClient by Nick O’Leary (v2.8.0)
• LiquidCrystal I2C by Frank de Brabander (v1.1.2)

Talking to the BME680

Getting the BSEC library set up is a bit of a pain. Once you’ve successfully got ESP32 code building and flashing, you’ll need to add the BME680 library to your IDE (Download the release as a zip from the above Github Link and add it to the Arduino IDE by selecting ‘Sketch’ from the menu bar, then head to ‘Include Library’, and then ‘Add .ZIP Library…’). Once you’ve done this, close out of the Arduino IDE, navigate to where your Arduino IDE saved the ESP32 platform (usually ~/Arduino/Hardware/expressif/esp32), and then open the platform.txt in a text editor (I suggest saving a copy elsewhere so you can restore if needed!). Look for a section that looks something like the below:

compiler.c.extra_flags=
compiler.c.elf.extra_flags=
#compiler.c.elf.extra_flags=-v
compiler.cpp.extra_flags=
compiler.S.extra_flags=
compiler.ar.extra_flags=
compiler.elf2hex.extra_flags=

At the bottom of this section, add this line:

compiler.libraries.ldflags=

Now that we’ve added ldflags, we need to reference them. Find the line that starts with recipe.c.combine.pattern and replace it with this:

## Combine gc-sections, archives, and objects
recipe.c.combine.pattern="{compiler.path}{compiler.c.elf.cmd}" {compiler.c.elf.flags} {compiler.c.elf.extra_flags} -Wl,--start-group {object_files} "{archive_file_path}" {compiler.c.elf.libs} {compiler.libraries.ldflags} -Wl,--end-group -Wl,-EL -o "{build.path}/{build.project_name}.elf"

Once you have done all of this, you can select one of the now available to you BSEC Example sketches (File > Examples > BSEC Software Library) and check it compiles and uploads to your board. Then consult the serial monitor to see if it works as you expected. You’ll probably find it doesn’t because nothing in life is easy and life is suffering. You should at least get an error code. If you haven’t, there is something more serious wrong. Here are a few pointers to get it working assuming you got a BSEC Error code. Well aren’t you lucky you at least get an error code! It’s nice to get an error code since you can go and look up the error code. In the documentation. Oh theres no error codes there? How about in the code? It’s closed source? Crap.

Firstly, you’ll most likely have to initialise the ‘Wire’ library with specific pins. This is easy enough, just find the Wire.begin() line and specify your pins in here:

# At the top of your sketch, below the #include lines
#define PIN_I2C_SDA 21
#define PIN_I2C_SCL 22
# Further down, in the setup() method
Wire.begin(PIN_I2C_SDA, PIN_I2C_SCL);

Secondly, my particular BME680 breakout board was for some reason configured to use the secondary I2C address, so I had to explicitly tell the BSEC library to use the secondary I2C address. Again, easy enough, find the line that contains BME680_I2C_ADDR_PRIMARY and replace it with BME680_I2C_ADDR_SECONDARY. Nice.

Talking to the LCD

This was simpler. Once again, you can head to the example code for the LCD to get a feel for what the library can do (File > Examples > INCOMPATIBLE > LiquidCrystal I2C) (the library doesn’t explicitly support the ESP32 but it seems to work fine for me!).

Example code I wrote - Arduair

Onto the good bit. I am not an Arduino expert, and I am not a C programmer, so I can’t promise this code is good. It works for me for now and I would like to tidy it up, but I present it in its current form below. It borrows code from both the BSEC Examples and the LCD examples. I’ve tried to document it with comments as well as possible.

// kn100.me - Arduairs
// Available on Github: https://github.com/kn100/arduairs
// Note: This code is provided as an example only - it does not implement authentication, TLS or anything even remotely close to security. It's probably fine if all your infrastructure lives on your local network, but you might want to consider looking into security.

#include <PubSubClient.h>

#include "bsec.h"
#include <LiquidCrystal_I2C.h>
#include <WiFi.h>
#include <EEPROM.h>

// Controls which pins are the I2C ones.
#define PIN_I2C_SDA 21
#define PIN_I2C_SCL 22

// Controls how often the code persists the BSEC Calibration data
#define STATE_SAVE_PERIOD  UINT32_C(60 * 60 * 1000) // every 60 minutes

const char* mqtt_server = "<your-mqtt-server-IP>";
const char* ssid = "<your-wifi-SSID>";
const char* passphrase = "<your-wifi-password>";

// Controls the offset for the temperature sensor. My BME680 was reading 4 degrees higher than another thermometer I trusted more, so my offset is 4.0.
const float tempOffset = 4.0;

// Configuration data for the BSEC library - telling it it is running on a 3.3v supply, that it is read from every 3 seconds, and that the sensor should take into account the last 4 days worth of data for calibration purposes.
const uint8_t bsec_config_iaq[] = {
#include "config/generic_33v_3s_4d/bsec_iaq.txt"
};

// Create an object of the class Bsec
Bsec iaqSensor;

String output;

// This stores a question mark and is eventually replaced with a space character once the BSEC library has decided it has enough calibration data to persist. It is displayed in the bottom right of the LCD as a kind of debug symbol.
String sensorPersisted = "?";

uint8_t bsecState[BSEC_MAX_STATE_BLOB_SIZE] = {0};
uint16_t stateUpdateCounter = 0;

WiFiClient espClient;
PubSubClient broker(espClient);

// The I2C address of your LCD, it will likely either be 0x27 or 0x3F
const uint8_t LCD_ADDR = 0x27;

// 20 characters across, 4 lines deep
LiquidCrystal_I2C lcd(LCD_ADDR, 20, 4);

void setup(void)
{
  Serial.begin(115200);
  broker.setServer(mqtt_server, 1883);

  Wire.begin(PIN_I2C_SDA, PIN_I2C_SCL);
  // Your module MAY use the primary address, which is available as BME680_I2C_ADDR_PRIMARY
  iaqSensor.begin(BME680_I2C_ADDR_SECONDARY, Wire);
  output = "\nBSEC library version " + String(iaqSensor.version.major) + "." + String(iaqSensor.version.minor) + "." + String(iaqSensor.version.major_bugfix) + "." + String(iaqSensor.version.minor_bugfix);
  Serial.println(output);
  checkIaqSensorStatus();
  iaqSensor.setConfig(bsec_config_iaq);

  //loadState here refers to the BSEC Calibration state.
  loadState();

  // List all the sensors we want the bsec library to give us data for
  bsec_virtual_sensor_t sensorList[10] = {
    BSEC_OUTPUT_RAW_TEMPERATURE,
    BSEC_OUTPUT_RAW_PRESSURE,
    BSEC_OUTPUT_RAW_HUMIDITY,
    BSEC_OUTPUT_RAW_GAS,
    BSEC_OUTPUT_IAQ,
    BSEC_OUTPUT_STATIC_IAQ,
    BSEC_OUTPUT_CO2_EQUIVALENT,
    BSEC_OUTPUT_BREATH_VOC_EQUIVALENT,
    BSEC_OUTPUT_SENSOR_HEAT_COMPENSATED_TEMPERATURE,
    BSEC_OUTPUT_SENSOR_HEAT_COMPENSATED_HUMIDITY,
  };

  iaqSensor.setTemperatureOffset(tempOffset);

  // Receive data from sensor list above at BSEC_SAMPLE_RATE_LP rate (every 3 seconds). There is also BSEC_SAMPLE_RATE_ULP - which requires a configuration change above.
  iaqSensor.updateSubscription(sensorList, 10, BSEC_SAMPLE_RATE_LP);
  checkIaqSensorStatus();

  lcd.init();
  lcd.backlight();
  // I wanted some iconography, so this function creates some icons in the LCDs memory. They were created using Maxpromers LCD Character Creator
  createLCDSymbols();
  connectToNetwork();
}

// Function that is looped forever
void loop(void)
{
  if (!broker.connected()) {
    reconnectToBroker();
  }
  broker.loop();
  // iaqSensor.run() will return true once new data becomes available
  if (iaqSensor.run()) {
    lcd.clear();
    displayIAQ(String(iaqSensor.staticIaq));
    displayTemp(String(iaqSensor.temperature));
    displayHumidity(String(iaqSensor.humidity));
    displayPressure(String(iaqSensor.pressure/100));
    displayCO2(String(iaqSensor.co2Equivalent));
    displayVOC(String(iaqSensor.breathVocEquivalent));
    displaySensorPersisted();
    updateState();
  } else {
    checkIaqSensorStatus();
  }
}

// checks to make sure the BME680 Sensor is working correctly.
void checkIaqSensorStatus(void)
{
  if (iaqSensor.status != BSEC_OK) {
    lcd.setCursor(0, 0);
    lcd.print("Sensor error");
    if (iaqSensor.status < BSEC_OK) {
      output = "BSEC error code : " + String(iaqSensor.status);
      Serial.println(output);
    } else {
      output = "BSEC warning code : " + String(iaqSensor.status);
      Serial.println(output);
    }
    delay(5000);
    lcd.clear();
    checkIaqSensorStatus();
  }

  if (iaqSensor.bme680Status != BME680_OK) {
    lcd.setCursor(0, 0);
    lcd.print("Sensor error");
    if (iaqSensor.bme680Status < BME680_OK) {
      output = "BME680 error code : " + String(iaqSensor.bme680Status);
      Serial.println(output);
    } else {
      output = "BME680 warning code : " + String(iaqSensor.bme680Status);
      Serial.println(output);
    }
    delay(5000);
    lcd.clear();
  }
}

// The below display functions display data as well as publishing it to the MQTT broker. They expect the area that they are rendering to to be free of characters.
void displayIAQ(String iaq)
{
  lcd.setCursor(0, 0);
  lcd.write(0);
  lcd.print(iaq);
  char carr[iaq.length()];
  iaq.toCharArray(carr, iaq.length());
  broker.publish("bme680/iaq", carr);
}

void displayTemp(String tmp)
{
  lcd.setCursor(14, 0);
  lcd.print(tmp);
  lcd.write(1);
  char carr[tmp.length()];
  tmp.toCharArray(carr, tmp.length());
  broker.publish("bme680/temperature", carr);
}

void displayHumidity(String humidity)
{
  lcd.setCursor(0, 1);
  lcd.write(2);
  lcd.print(humidity + "%");
  char carr[humidity.length()];
  humidity.toCharArray(carr, humidity.length());
  broker.publish("bme680/humidity", carr);
}

void displayPressure(String pressure)
{
  lcd.setCursor(12, 1);
  lcd.print(pressure);
  lcd.write(3);
  char carr[pressure.length()];
  pressure.toCharArray(carr, pressure.length());
  broker.publish("bme680/pressure", carr);
}

void displayCO2(String co)
{
  lcd.setCursor(0, 3);
  lcd.print("CO2 " + co + "ppm");
  char carr[co.length()];
  co.toCharArray(carr, co.length());
  broker.publish("bme680/co", carr);
}

void displayVOC(String voc)
{
  lcd.setCursor(0, 2);
  lcd.print("VOC " + voc + "ppm");
  char carr[voc.length()];
  voc.toCharArray(carr, voc.length());
  broker.publish("bme680/voc", carr);
}

void displaySensorPersisted()
{
  lcd.setCursor(19,3);
  lcd.print(sensorPersisted);
}

void connectToNetwork() {
  lcd.setCursor(0, 0);
  lcd.print("Connecting to");
  lcd.setCursor(0, 1);
  lcd.print(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, passphrase);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Establishing connection to WiFi..");
  }
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print("Connected");
  Serial.println("Connected to network");
  delay(1000);
  lcd.clear();
}

void reconnectToBroker() {
  // Loop until we're reconnected
  while (!broker.connected()) {
    lcd.setCursor(0, 0);
    lcd.print("Connecting MQTT");
    lcd.setCursor(0, 1);
    lcd.print(mqtt_server);
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "aqm-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (broker.connect(clientId.c_str())) {
      Serial.println("connected to broker");
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Connected");
      delay(1000);
    } else {
      lcd.setCursor(0,2);
      lcd.print("Broker failed");
      Serial.print("failed connecting to broker, rc=");
      Serial.print(broker.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
    lcd.clear();
  }
}

void createLCDSymbols() {
    byte iaqsymbol[] = {
    0x04,
    0x0A,
    0x1F,
    0x11,
    0x0E,
    0x0A,
    0x0E,
    0x02
  };
  lcd.createChar(0, iaqsymbol);
  byte tempSymbol[] = {
    0x18,
    0x18,
    0x07,
    0x04,
    0x04,
    0x04,
    0x04,
    0x07
  };
  lcd.createChar(1, tempSymbol);
  byte humiditySymbol[] = {
    0x04,
    0x04,
    0x0A,
    0x0A,
    0x11,
    0x11,
    0x0A,
    0x04
  };
  lcd.createChar(2, humiditySymbol);
  byte airPressureSymbol[] = {
    0x04,
    0x15,
    0x0E,
    0x04,
    0x01,
    0x1E,
    0x01,
    0x1E
  };
  lcd.createChar(3, airPressureSymbol);
}

// loadState attempts to read the BSEC state from the EEPROM. If the state isn't there yet - it wipes that area of the EEPROM ready to be written to in the future. It'll also set the global variable 'sensorPersisted' to a space, so that the question mark disappears forever from the LCD.
void loadState(void)
{
  if (EEPROM.read(0) == BSEC_MAX_STATE_BLOB_SIZE) {
    // Existing state in EEPROM
    Serial.println("Reading state from EEPROM");

    for (uint8_t i = 0; i < BSEC_MAX_STATE_BLOB_SIZE; i++) {
      bsecState[i] = EEPROM.read(i + 1);
      Serial.println(bsecState[i], HEX);
    }
    sensorPersisted = " ";
    iaqSensor.setState(bsecState);
    checkIaqSensorStatus();
  } else {
    // Erase the EEPROM with zeroes
    Serial.println("Erasing EEPROM");

    for (uint8_t i = 0; i < BSEC_MAX_STATE_BLOB_SIZE + 1; i++)
      EEPROM.write(i, 0);

    EEPROM.commit();
  }
}

// updateState waits for the in air quality accuracy to hit '3' - and will then write the state to the EEPROM. Then on every STATE_SAVE_PERIOD, it'll update the state.
void updateState(void)
{
  bool update = false;
  /* Set a trigger to save the state. Here, the state is saved every STATE_SAVE_PERIOD with the first state being saved once the algorithm achieves full calibration, i.e. iaqAccuracy = 3 */
  if (stateUpdateCounter == 0) {
    if (iaqSensor.iaqAccuracy >= 3) {
      update = true;
      stateUpdateCounter++;
    }
  } else {
    /* Update every STATE_SAVE_PERIOD milliseconds */
    if ((stateUpdateCounter * STATE_SAVE_PERIOD) < millis()) {
      update = true;
      stateUpdateCounter++;
    }
  }

  if (update) {
    iaqSensor.getState(bsecState);
    checkIaqSensorStatus();

    Serial.println("Writing state to EEPROM");

    for (uint8_t i = 0; i < BSEC_MAX_STATE_BLOB_SIZE ; i++) {
      EEPROM.write(i + 1, bsecState[i]);
      Serial.println(bsecState[i], HEX);
    }

    EEPROM.write(0, BSEC_MAX_STATE_BLOB_SIZE);
    EEPROM.commit();
  }
}

Server Software

MQTT

If you’re using the code above, you’re going to need a MQTT broker somewhere to publish to. Assuming you have a Raspberry Pi or something set up on your local network, you want to install Mosquitto. Once you’ve confirmed you can publish and subscribe to Mosquitto, you can put the IP of the MQTT broker into the code above. You should then be able to run the ESP32 code above and have it start displaying stuff to the LCD and from another machine on your network that has the Mosquitto client installed do

mosquitto_sub -h <mqtt-broker-ip-address> -t "bme680/#"

and have it return the same data as is displayed on the LCD!

From this point, you could install an app on your phone like MQTT Dash which you can then configure with details of your MQTT broker so you can display your air quality data in real time on your phone.

Persisting data with Influxdb

I wanted persistence, and that’s where Influxdb comes in. Install Influxdb on your server and confirm that you can access your Influxdb instance (the influx command should result in an Influxdb shell).

Now you’ll need some way to:

1. Subscribe to your MQTT broker and receive events relating to the bme680 topic
2. Connect to Influxdb
3. Write this point data to Influxdb.

I provide example code I wrote in Golang below. it is also available on Github - See the mqtt680influxbridge project

package main
// kn100.me - Arduairs
// Available on Github: https://github.com/kn100/mqtt680influxbridge
// Note: This code is provided as an example only - it does not implement authentication, TLS or anything even remotely close to security. It's probably fine if all your infrastructure lives on your local network, but you might want to consider looking into security.

import (
	"fmt"
	"log"
	"os"
	"os/signal"
	"strconv"
	"syscall"
	"time"

	mqtt "github.com/eclipse/paho.mqtt.golang"
	influx "github.com/influxdata/influxdb1-client/v2"
)

func main() {
	var (
		mqttAddress     = envString("MQTT_ADDRESS", "127.0.0.1")
		mqttPort        = envString("MQTT_ADDRESS", "1883")
		mqttTopic       = envString("MQTT_TOPIC", "bme680/+")
		influxDbAddress = envString("INFLUXDB_ADDRESS", "127.0.0.1")
		influxDbDb      = envString("INFLUXDB_DB", "bme680")
	)

	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)

	influxHost := fmt.Sprintf("http://%s:%d", influxDbAddress, 8086)

	influxClient, err := influx.NewHTTPClient(influx.HTTPConfig{Addr: influxHost, Timeout: 5 * time.Second})
	if err != nil {
		log.Fatal(err)
	}

	opts := mqtt.NewClientOptions().AddBroker(fmt.Sprintf("tcp://%s:%s", mqttAddress, mqttPort))
	mqttClient := mqtt.NewClient(opts)
	if token := mqttClient.Connect(); token.Wait() && token.Error() != nil {
		panic(token.Error())
	}

	var f mqtt.MessageHandler = func(client mqtt.Client, message mqtt.Message) {
		log.Printf("received message on topic: %s - message: %s\n", message.Topic(), message.Payload())
		topic := message.Topic()
		val, err := strconv.ParseFloat(string(message.Payload()), 32)
		if err != nil {
			log.Println("invalid point data received from broker, ignoring")
			return
		}
		fields := map[string]interface{}{
			topic: val,
		}
		influxPoint, err := influx.NewPoint(message.Topic(), nil, fields, time.Now())

		bp, err := influx.NewBatchPoints(influx.BatchPointsConfig{
			Database:        influxDbDb,
			Precision:       "s",
			RetentionPolicy: "30_days",
		})

		if err != nil {
			log.Fatalln("Error: ", err)
		}

		bp.AddPoint(influxPoint)
		err = influxClient.Write(bp)
		if err != nil {
			log.Println("couldn't write to influx for some reason - ignored", err)
			return
		}
		log.Println("data written to influx")
	}

	if token := mqttClient.Subscribe(mqttTopic, 0, f); token.Wait() && token.Error() != nil {
		log.Fatal(token.Error())
	}
	<-sigs
	log.Println("Exiting")
}

func envString(key, fallback string) string {
	if s := os.Getenv(key); s != "" {
		return s
	}

	return fallback
}

This code is the most basic possible bridge I could think of. It immediately writes every data point that it gets from the subscription to Influx, completely ignoring the helpful batching logic that the library provides. A significant improvement to this code would be to do batching properly. I couldn’t be bothered for now, maybe in a future blog post.

As soon as you’ve gotten this code logging that is is writing to Influx, it’s time to move onto Grafana.

Look at this graph - Grafana

Grafana is nice. You give it a database connection, it’ll provide you with a nice visual query builder that you can use to build graphs. You probably want to install Grafana on the same instance that is hosting your Influxdb. Here’s a nice tutorial on getting Grafana set up.

I provide my Dashboard model below, which you are free to take and import into your Grafana instance. You’ll firstly need to add your Influxdb as a data source to Grafana. On the left, select the cog wheel, and then Data sources. Then select ‘Add data source’ and configure exactly as below other than the URL (this should either be localhost or the host where your InfluxDB instance lives.

Then create a new dashboard. On that blank dashboard, select the cog wheel in the top right, and paste this in JSON Model. when you head back to your dashboard, you now should see some pretty graphs that look somewhat like what I promised.

Let me know how you got on!

I installed YTMDesktop using Snap - a tool I don’t fully understand nor like very much. I installed it using Snap since as far as I can tell it’s a nice way to install apps you don’t care that much about and supposedly provides some security niceties. After I figured out that the stuttering only happened while Youtube Music was playing music, I erroneously reported the bug to the Gnome Shell team. I even attempted to record the issue using a screen recording tool, but the stutters were bad enough that the screen recording tool itself was stuttering too and therefore not picking up on the lost frames! Almost immediately someone there pointed out it might be related to AppIndicator - so I tried disabling it through the inbuilt extensions menu in Pop!_OS (but you can install the Gnome Tweak Tool if this isn’t available for you!) the stuttering immediately stopped! You can see this for yourself if you’re affected by using something like the Blur Busters UFO test.

I dug a little deeper and found in my journalctl there were hundreds of entries from YTMDesktop where it was attempting to access some resource in a flatpak related directory!

May 18 13:39:15 pop-os audit[36663]: AVC apparmor="DENIED" operation="open" profile="snap.youtube-music-desktop-app.youtube-music-desktop-app" name="/home/kn100/.local/share/flatpak/exports/share/icons/hicolor/48x48/apps/" pid=36663 comm="youtube-music-d" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

It seems like maybe (and I do not understand Snap or Flatpak at all - although both are in use on my system) the Youtube Music Desktop App was originally packaged for Flatpak and then stuffed into a Snap. It’s trying to access nonexistent resources now which assumedly is why AppIndicator is crashing.

It seems absolutely crazy to me that an extension to Gnome Shell could cause such a UI disaster - and wonder how many other apps cause similar issues. Users everywhere might install one of these apps and decide that Gnome Shell is garbage, as I have a few times in the past. From a little Googling, it seems that Discord and Megasync are both also affected. Discord seems a little egregious given that it’s mainly used while gaming - an activity where stuttering is unacceptable.

For now I have disabled AppIndicator - but now certain applications I use have significant usability issues. When you close them, they remain open in the ‘system tray’ that no longer exists, and therefore you have no way to access them now!

I understand the Gnome teams design goal of getting rid of system icons, but given that basically every OS packages AppIndicator or something similar since some apps absolutely require you to have access to their tray icon - it seems like maybe this design goal has hit the rails of reality and needs to be reconsidered.

I’ve added my issue to the already existing issue on Github here.

One of the biggest lessons I learned was to prioritise ‘velocity’ over ‘speed’. This probably sounds meaningless to you, but allow me to explain. At some point around a year ago, during my one to ones with my manager, we were discussing the fact that I was heavily distracted and finding it tough to concentrate on tasks I wanted to work on. We established that one of the main causes of this was the fact that I would agree to help anyone who happened to ask for my help. Sales has an issue with something? Yes sure I’ll look now! Marketing wants some data on something else? Why not, I’ll look now. A more junior engineer would like help in a particular area of the codebase that I happen to be more familiar with? Sure, no problem, lets pair! I had high ‘speed’. I was helping others to accomplish their goals with no consideration for how to prioritize tasks to provide my team and the wider company with the maximum value I could provide.

Learning to prioritize, knowing when and how to say no, and most importantly when to say yes are crucial skills in improving your ‘velocity’. Velocity in this sense is being used to describe the pace at which you are delivering your maximum value. By constantly saying yes to others requests for assistance, I was failing to perform a value judgement comparing whether helping them to continuing with the task I was originally doing. As we all know, context switching is expensive. This led me to be known as one of the most helpful people at the company (I had more high fives than anyone else at the company!), but I also became known for being constantly busy and giving bad time estimates.

I believe it is innate in most of us to be helpful, however being helpful can actually reduce your overall effectiveness as an engineer. The best advice I can offer on this is best given in the form of a proverb of highly contested origin: “Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime.”.

Code bases that have been around for a while might look weird, but they might just be solving problems you’re not yet aware of.

I’m sure every single fresh-out-of-university experience-lacking engineer has had that moment where they join a company and see the code that powers it for the first time. Seemingly simple routines seem more complex than they need to be. Seemingly small problems have huge solutions. Craziness, in short. It can be terrifying. You’re fresh out of school, looking at this codebase, wondering what the heck is going on? Why is everything so complex?

It turns out that managing a huge codebase is a hard problem, and a junior engineer is rarely a good judge of the scale or scope of a given problem. It’s not that there isn’t bad code in the codebase, its just more likely that any junior engineer with 3 months on the clock has most likely not seen enough of the system to fully understand the problems it is solving.

As time went on, I learned more about the actual problems this codebase was addressing. What I perceived as overcomplications turned out to be problems I wasn’t aware of being solved. What I thought was badly designed code was actually just code I didn’t understand. It’s intimidating to work on such a large system, especially when the most ambitious project you’ve worked on prior to it was several orders of magnitude smaller and not subjected to the brutality of customers, but if you’re anything like me, you’ll learn ridiculously quickly.

Team culture is a real not-bullshit thing.

By far my favourite thing about my first job was the learning opportunities around me. I was surrounded by massively intelligent, kind, determined people who all were more than willing to help me get up to speed.

I believe this environment didn’t happen accidentally. The company was very careful to create a culture where this kind of collaboration could happen. Much more senior engineers spent probably weeks of their lives showing me the ropes and teaching me about anything and everything I wanted to know, work related or not. Every single one of these people was hugely passionate about what they do and loved to share their knowledge.

Outside of work, these people held clubs and sessions relating to areas of interest to them personally. I was particularly fond of but sadly nowhere near involved enough in an electronics session - the last activity of which was to build Tetris with nothing but ’nand’ logic gates! This environment is perfect for a junior engineer to thrive in, and I am hugely grateful to have had a job where the conditions were so ripe for it.

As I became more and more established and newer members of the team were hired, I found myself in a position to pay it forward, if you will. What I learned during these encounters was that the educational aspect of it goes both ways. The field is such a vast one that everybody probably knows something somebody else doesn’t, so learning new things from these engineers was a common occurrence for me. The learning wasn’t always technical either. I developed people skills I didn’t have before. I began delegating. I grasped the best attitude to have towards people and the problems we faced together.

You, as much as anyone else, foster this spirit of reciprocal learning. Represent and grow it the best way you can.

Look for opportunities to lead a project, and just do it.

The average engineer at a startup seems to have to wear many hats. I found myself working on the core of the system in one language and on the billing system in another within the same week. I maintained SDKs in multiple different languages. I worked on our frontend as well as backend. This is a natural requirement at a startup; there just aren’t enough people to cleanly define peoples roles and have them stick to them. I think its fair to say anyone in this position will gain a diverse set of skills that don’t necessarily join up neatly. It also provides many opportunities for identifying problems with areas of the system that others just haven’t looked at recently.

These are amazing opportunities for the ambitious engineer. It is possible to pick one of these areas and become the leader of a project in a startup environment.

If you wish to succeed in convincing your team that the problem you’ve identified is worth solving, you’re going to want to become the domain expert on the topic. Read up on other solutions to the problem you’ve found. Talk to other engineers in the company about whether they’re aware of it or even if they’ve attempted to solve it before. Quantify the problem. The most effective metrics I’ve found are monetary savings or developer efficiency improvements. The question you should be answering is why it is valuable for you to spend your time on this project rather than something, or someone else. You’re not solving a problem; you’re instead executing on an opportunity you’ve identified.

It is important to remember here though not to outgrow your boots. This fresh into the field you can easily find yourself chasing rabbits and trying to solve huge problems that much more experienced engineers would struggle with. The best opportunities are the low hanging fruit.

Imposter syndrome is real, and it sucks.

I’m not going to write much about this since so much has been written on the topic already. In my experience everybody suffering from it likely thinks that they’re not the ones suffering with imposter syndrome; they’re actually imposters! They’re somehow flying under the radar, holding on by the skin of their teeth.

This is most likely not the case. Think about it logically. This field is generally well paid. The company is paying you for your time. In order for it to make sense for them to do so, you must be delivering value to the company in one form or another. You’re probably not special. You’re not fooling anyone. You’re a junior engineer who has probably passed through their probationary period and is doing their job well enough to warrant the money you receive.

In the first few months, focus on a year from now.

At a startup, you’re likely being brought on to wear many hats. Different kinds of work require different levels of context and different levels of experience. Its natural to gravitate towards tasks that require lower context and little experience, since these are the ones you’re most likely to deliver results on quickly. If you’re like me, you’ll feel that for the first few months you aren’t really delivering all that much value to the company, and you may even feel a little guilty about it. Why are they paying me to sit here and not know what I’m doing?!

It turns out that you aren’t the first software engineer hired fresh out of university, and they’re as aware as you are of your lack of experience. There’s no reasonable way you could contribute immediately. It takes time to acclimate to the ‘real world’ and to learn the code and company well enough to be able to deliver quickly.

Don’t make the same mistake I did and just focus on delivering as much as possible as early as possible. This pays off early but hurts you in the long run, since you’re not giving yourself the chance to properly dig into the code. Spend this early time figuring out what interests you in the code and how you can deliver value to yourself while delivering for the company. This might involve learning a type of system you’ve never seen before, or a programming language you didn’t know before. This is okay! That is what this period is for. You’re focussing on your own personal development in this time so that later you can deliver significantly more value.

If you instead continue on the path of delivering as much as possible as quickly as possible, you’ll reap the rewards early on but you’ll probably lose motivation as you continually work on projects that don’t get you excited.

Own your fuckups. They’re probably okay.

Every engineer is likely to do something that negatively impacts other engineers or customers. I personally recall accidentally leaving debug logging on a service which caused that service to use much more CPU and run slower for a bit. Thankfully we had CPU to spare, but this really shook me up. What surprised me was the completely accepting atmosphere I found myself in. I wasn’t told off, and instead was asked to look into making a mistake like this impossible to repeat. This meant what was a very minor mistake with no impact to anything really except the heat output of a data center turned into a real reliability improvement - since mistakes like this could very well turn into very serious issues if not mitigated.

I get that not every work environment is going to be like this, but if yours is, lucky you. Go forth, make new mistakes.

Productivity is not what you likely think it is.

Productivity in Engineering is really difficult to quantify. Lines of Code is not a good measure. Neither are commits. Your days where you write no code can be massively productive.

This may sound obvious, however it took a conversation between me and a colleague, Jim to realise this.

One day a year or so ago, I was having an extremely hectic day. I was all over the place. I was pairing with Engineers who had just joined the team to get them up to speed. I was writing docs. I was managing a contractor. I was participating in meetings As the day wore on, I became more and more aware of the fact I’d made very little progress towards the goals I’d set myself. I’d written zero code.

I ranted to Jim about how I felt like I had got nothing done. He then told me in no uncertain terms that if all I was valued for was LoC, I’d be a code monkey, and that in fact I’d had a hugely productive day. This conversation made me think. Other colleagues who I thought of as massively productive rarely wrote code too! Maybe I really was productive!

Communication is HARD.

I’d like to think that I am a good communicator. I have the gift of the gab and will talk for Wales if given the opportunity. I’m well known for being immensely friendly and constantly happy. I found regularly that I was misunderstood however. How could that be? This job taught me that it is possible to over-communicate. I was talking a lot, but not conveying much meaningful concrete information. When it came to collaborating with other engineers, I was great at talking about what we were doing but terrible at co-ordinating the work we were doing.

The way I began to solve this was pretty simple. I decided that any concrete decisions needed to be communicated in as few words as possible via writing. Anything more visionary could be discussed in person, but the outcomes needed to be written down. If there was stuff to be done, I assigned people to action them. If there was stuff that needed to be discussed, I scheduled a meeting and had an agenda. Not exactly rocket science, but an extremely useful lesson to learn.

This led to less words, more effectively spoken.

Giving talks is a great way to bring up others perceived value of you.

This links closely with the communication point I discussed above. I started giving talks to introduce others to what I was doing, and to open lines of communication that weren’t previously there.

One time, I gave a talk on a project I was working on related to Terraform. I was working on importing AWS resources into Terraform so that we wouldn’t have to recreate infra. This task was tricky, and involved many fiddly parts that didn’t quite work properly. Terraform is not really suited to import massive amounts of old infra. After the talk, a member of another team offered his assistance to me as he’d been working on Terraform recently too. I had no idea that others had experience with Terraform, and his assistance proved invaluable.

Other ways this is incredibly useful is to get the rest of an organisation excited about what it is that you do. It’s all well and good having your direct team-mates understanding the value of your work, but what about the salespeople? Marketing? The quickest way to pique their interest in my estimation was a talk.

When giving a talk to people who aren’t directly related to the tech you’re working on, it’s important to remember they likely don’t care that much or understand that well the “how”. They want the “what”, “why”, “who”, and “when”. Put yourself in the shoes of the audience member you most want to get through to. Ask these questions.

• What is this work to me?
• Why should I and those I communicate with care about this?
• Who does this benefit?
• When is it available?

Your talk doesn’t even necessarily need to be about what you’re doing. I gave a talk once on Silicon bugs. This was pretty irrelevant to the day-to-day, but people found it very interesting and that sparked a few conversations with people around the business I had never spoken to before. These cross-business links can prove invaluable.

At some points, you might be the most knowledgeable.

This is a scary realisation. As you dig more and more into the problems you’re solving, you’ll begin to specialise in particular areas of the product. As a junior engineer, you’re likely used to the comfort of having a senior engineer behind you to tell you when you’re doing well and when you’re not. Your specialties in understanding develop, and suddenly your senior engineer tells you that they don’t know! They’ve got useful general advice, but whether your approach is correct or not, they don’t know. You look up, and there is nobody to defer to now.

At this moment, you are probably the most knowledgeable on the topic at hand. You become responsible for seeking the answer.

Even more scary is when more junior engineers than you start to look up to you in the way you looked up to your senior engineer in the past. You look up again, and there’s nobody to ask for specifics! This is when you try to pass on to the junior engineers useful ways of figuring out the answer that you developed when you first realised there’s nobody to look up to anymore. These can range from simple “how to use Stackoverflow” to more specific “what’s enough due diligence to verify you’re not going to break production?”.

Disclaimer: This is mostly opinion, and only my opinion. This post is not associated with my employer in any way. You can contact me on Mastodon @kn100@fosstodon.org

What even is a breaking change, anyway

A simple definition of a breaking change is any change you make to your code that could break other code which directly or indirectly depends on it.

In all seriousness, almost anything you do within a Go library that others depend on could be considered a breaking change. Did your commits fix a bug that was reported, without changing the API? It’s possible there is code out there that was depending on the presence of that bug, and you’ve just broken their code. Did you update a dependency you depend on? Did your dependency fix a bug consumers of your library were depending on? You get the point.

For the purposes of this article, I align my definition of a breaking change closely with the one given by Semver.org, which is any “incompatible API change”. The key word there is API. This (to me at least) indicates that bug fixes or similar would be considered patch or minor version updates. If you anticipate that you’ll break someones code through changing your API, that’s a major version increment. Otherwise, it’s a minor or patch. The clue is in the word “semantic”. A Semver number is intended to be read by humans, not computers. This does not guarantee that a minor version increment is going to be absolutely fine for everyone. Those edge cases where consumers rely on the presence of a bug are difficult to anticipate and therefore are not considered here. The more significant an increment, the more trouble the user is likely to have when upgrading.

A story on making a breaking change

I maintain a library that is written in Golang. It’s a pretty standard library, and has a pretty simple API associated with it, maybe 7-8 functions that consumers can call. Nothing specialist. It was a library that had some small design issues as well as old code that was now dead. I’d been planning to make a breaking change in the API for a few months, and waiting for an opportune time to do it. In total there were 3 distinct breaking changes I wanted to make, only one of which was really ‘required’, the others were stylistic/linter improvements that I wanted to make.

I made three separate pull requests, had them approved by others who maintain said library, and made three separate releases on the same day, bumping the ‘major’ version number three times on the same day. I respect that these could have all been rolled into one major version bump, but I personally did not see the harm in incrementing the major version. This looked a little ridiculous, but my thinking was that this would allow users to upgrade incrementally, rather than having to deal with 3 conceptually different breaking changes in one go.

I thought this was enough to indicate that changes had taken place. I tried to describe exactly the changes that were made - and give clear instructions as to what to do to migrate up to that version.

What I had of course forgotten about is that Go ideally does not expect you to ever make breaking changes. A few of the libraries consumers rightly complained about the breaking changes, which is what inspired this article.

“You changed AppId to AppID and also UserId to UserID and it took me hours to figure out why my deployments weren’t working. Come on guys lets not introduce breaking changes like this without some sort of heads up!”

Go get - somewhat flawed

In the very early stages of development, when a consumer of a library wishes to consume it, they’ll usually use go get, a tool which requires basically no thought to use. It will fetch whatever is on master at that point. Not what you’ve released, and not whatever is on a release branch. go get is pretty much git clone, with Go tendencies.

The consumer will then eventually run go build or go run and their software will build and run as normal, even if the library author makes a breaking change. The Built binary is, well, built, so it will be fine. The developers build environment will be fine since it effectively vendors the library code - and won’t update it unless asked. If the user dares do a go get -u on the package, or recreates their build environment locally, they’ll receive the breaking changes, and therefore will have to deal with them.

The trouble starts when that consumer then either tries to build their code on another machine. The consumer go gets the packages they need again, except now there has been a slew of commits merged into libraries master branch, therefore inexplicably breaking their build.

The user then spends a few minutes trying to figure out why what works on their machine does not work in their CI environment, or on their AWS instance, or in their Docker container. Eventually, they find log lines that indicate a variable has been renamed, or a functions return type has changed, or the function they were using is gone altogether!

Frustrated, they either stop using your library, or they contact you and ask why you made such a change.

This whole process was designed by Google, for Google. It was designed for how they manage their repositories. Of course, other organizations manage their source code differently, and conflicts were bound to happen. This lack of package management meant that breaking changes rendered once fine builds completely broken. In other languages where package management has been a first class citizen for longer - this is not an issue, but in Go, it is. No amount of Semver is going to dig us out of this hole.

The fundamental problem with Go Package Management

I am not going to talk too much about the state of package management in Go. Enough has been said on that topic already. What I will say is none of them really address this problem. Not really. It doesn’t matter how good your package manager is. It makes no difference whether it vendors your dependencies. It is not the default, and therefore breaking changes are going to break someones code.

Since anybody writing go today isn’t likely to want to think about the intricacies of dependency management immediately, and it is certain that there are projects everywhere that do not use a package manager, it is crucial to consider the various factors that weigh into whether a breaking change is worth it or not.

The problem is that you are going to have consumers who are going to consume your code and expect there to never be a breaking change. This is the unchangeable reality of Go.

Are breaking changes possible then? This is a question that only the owners of the library can answer. How many consumers do you have? Does the benefit of making the breaking change outweigh the almost certain annoyance your consumers will face? Are there ways of making the change without breaking the build? Do you have a mechanism to warn users that breaking changes are coming? The factors which weigh in are numerous and each should be considered.

So, How DO I make breaking changes

There are many approaches to making breaking changes in Go. This is part of the problem. Without unifying our approach as library maintainers, we’ll struggle. I have no idea what the correct approach should be, and there are more than are listed below, but I hope this brief survey will be useful when you make your decision as to how to do it.

Just never making breaking changes ever

This is the only real solution. If not annoying users is your absolute top priority, then never make breaking changes! Design your API upfront to be as close to perfect as possible, and only ever add code that does not change the API.

This is the approach that many crucial components of software take. Only ever add functionality, never take it away or change existing functionality. Never break the build. This approach ensures you’ll annoy the fewest people, but of course has the disadvantage that you’re unable to fix any mistakes in your original spec.

Branch off for breaking changes

Another approach that some libraries take is to maintain version branches. For example, see github.com/robfig/cron. Here we can see there is a master branch which I assume v0, and then branches for each breaking change. I expect that the idea is that a user who wants to ensure their code will build forever will move to the directory where Go checked out the code, and then move to the branch that their code is written to support.

This ensures that (as long as the library author keeps their promises!) users who remember to check out the correct branch can expect their builds to continue working and their code to continue working.

This has the same downside that users do not have an obvious way to know how to move up major versions, and puts the onus completely on them to determine if there has been a major version update. If they decide to update major versions on one build machine, they must then follow suit everywhere else. This isn’t great.

Somehow notifying consumers of breaking changes

I’ve heard suggestions that maybe library authors should notify consumers about upcoming breaking changes. One approach is to release a version of your library where you log out that a given thing is changing on a certain date, and on that date release the next version which actually removes or modifies said functionality. This approach relies on the user consulting their logs frequently for these notices - something I’m not particularly hopeful happens. I expect most developers including myself only ever really consult logs when things go wrong, and here they go wrong when the thing is changed, meaning the log lines helped nobody. Semver seems to advocate for this approach, suggesting that we “issue a new minor release with the deprecation in place”. What form this deprecation takes however is left undefined.

Another suggestion I heard from other developers was to somehow contact the consumers of the library to inform them that the breaking change is coming. For instance, users who care about breaking changes could sign up to a mailing list. Maybe you have a method of identifying and contacting consumers already. Then, when you anticipate a breaking change, you contact the customer via email or something to let them know. This approach has a similar problem to the first, which is that I expect people signing up to said mailing list would only do so as a reaction, rather than proactively. Contacting your consumers with emails you’ve already got somewhere else somewhat solves this, but will users really read this email? Will they action it? Will they even realize that the change affects them?

Declaration that breaking changes are to be expected

Given that almost every library that is not already well defined is likely to want to change its API at some point, I think the most realistic approach we’ve got is to explicitly declare that breaking changes are to be expected in the README. Something along the lines of:

This library, and any others that you are using, may make breaking changes at any time. These changes could unexpectedly break your build. To avoid this, we recommend you use a package manager such as the inbuilt Go Modules, a tutorial for which can be found here.

This is clearly not perfect. This requires your consumers to take action and do something in order to prevent this problem in the future - however I think it is fairly reasonable. If we look at other examples of how this problem is handled in other languages, we’d see that libraries sometimes even include a snippet to copy into the Gradle file or the language itself will write a package lock file that locks to whatever the latest version was at the time of installation.

This does however point Go consumers in the correct direction for safely using their libraries, and even better, thanks to the way Go modules works, even solves the problem automatically for other libraries that are installed! This doesn’t solve the problem at all for consumers that will not or cannot use a package manager, so this, in use with a commitment to make breaking changes as infrequently as possible seems like a solution that will annoy the fewest people over time.

In my humble opinion, use of a package manager should become the norm. Bare go get should be discouraged for everything except experimentation. I have been bitten by others making API breaking changes and I personally have inflicted that pain on others. It’s an easy thing to miss - and there are many situations where using a package manager isn’t ideal - but in a world where breaking changes are going to be made, pretending that they’re not going to be is not a solution.

Go mod has its own idiosyncrasies, but it’s as close to a built in package manager Go has gotten so far. I think it’s worth it in the long run for libraries to encourage its use and to make it the new normal for developers just starting out in Go. The thousands of unbuildable projects floating around that have depended on libraries that broke backwards compatibility will look on and smile.

I tried a few different things initially. I firstly tried hosting a Minecraft server on a few different ‘free’ VPS suppliers. Immediately, the lack of RAM was an issue, creating unplayable lag spikes that took the fun out of it. It took me a while to realise that there was no free lunch, and nobody was going to host my ridiculously resource hungry Java based server in a way that would be beneficial to me. Gigabytes of RAM were expensive.

Next, I tried hosting it locally, but kept hitting up against a concept my young brain knew nothing about - which was port forwarding. I knew it required access to the Router, but that was guarded by my father, who was understandably and correctly worried about letting his son mess about in settings that he himself also did not really understand.

Furious, rushed Googling led me to Hamachi. Hamachi, now owned by LogMeIn is a hosted VPN service. Effectively it puts you and your friends who want to play on a VPN together. This worked a little better than my previous attempts, but still introduced a lot of latency, and as such wasn’t ideal. I stuck with this for a little while, but there had to be a better way.

The only other time I’d ever come across port forwarding was in my use of Bittorrent. Supposedly, without an open port, Bittorrent couldn’t function correctly. How did it function correctly then, I wondered? My client of choice (Transmission) would dutifully tell me whichever port I selected was ‘open’. Were all the ports open? Was my Minecraft server lying to me? After all, Transmission seemed to work perfectly downloading and even seeding things. I regularly used Bittorrent for distributing files between my few PCs and friends at the time, and it seemed to work perfectly.

It didn’t seem to matter which port I selected, when I pressed that “Test Port” button, somehow it was open! This wasn’t fair! How come the router stazi allowed Transmission to talk to whoever it wanted to, whenever it wished, yet my poor Minecraft server received no such special treatment? Was Big Torrent bribing Netgear? What sort of nepotism was this?!

UPNP meant nothing to me, and I didn’t think to Google it at the time, but it was the technology that allowed Transmission to talk to whoever it wanted.

One day, almost by accident, I found a solution, that was far better than Hamachi. It turned out that if I opened the Minecraft Server first, then entered the port that it was configured to use into the Port box within Transmission, and then hit Test Port, and left Transmission open while me and my friends would waste hours building things I’d eventually accidentally blow up with TNT, it just seemed to work!

What I didn’t realise back then was I was taking advantage of a security flaw within UPNP. Even though I was innocently hijacking the port Transmission opened for use with Minecraft, I was actually exploiting a fairly serious security vulnerability in the protocol. UPNP allows any old software to punch a hole through your routers firewall. It doesn’t verify which software is making use of said port. Any old code running on your computer is able to do this, including most terrifyingly to me at the time I realised what I was doing the hugely popular Adobe Flash.

How UPnP works

UPnP is a complicated standard whose overarching goal is to allow devices on a network to discover each others presence without require configuration from the user. It seems to want to abstract over various network configurations to reduce the mental burden of the user using a product which supports it. A noble goal. Far too much to talk about here! Instead I will focus on one part of UPnP, which is how it allows devices inside your network to punch holes through your routers firewall (port forward).

UPnP makes use of Internet Gateway Device Protocol, which is a protocol for mapping ports where NAT (Network Address Translation) is in use. Effectively it allows devices on a network to communicate their requests for a router to route data hitting a particular port to a the system making the request. IGDP also allows applications to learn the external IP address without having to use an external system, and even see which port mappings already exist. Crazily, there is even provision for the standard to allow applications on your network to request the router get a new public IP address, however I’ve personally never seen this in use.

When an application, for example Transmission, wants to map a port for its use, it makes a request to what the RFC for this refers to as the IGD Control point (in most cases the router).

After this, and if authentication is either not enabled or passes (it is not enabled in most cases!) - the IGD sends a “AddAnyPortMapping()” request through to the Internet Gateway Device - Port Control Protocol Interworking Function, or the IGD-PCP IWF for short. Oh aren’t RFCs fun! The IGD-PCP IWF then makes a Port Control Protocol MAP request to the PCP server.

The PCP Server can then do one of three broad categories of things:

• It can refuse. It may refuse because the port has already been opened for someone else. It may refuse for other reasons too.
• It can accept, and return a PCP MAP Response with the assigned external port the UPnP Control Point requested
• It can accept, and return a PCP MAP Response with another external port that the UPnP Control Point did not request but it can use if it wishes.

The application that triggered this whole flow may then test that the port mapping worked - usually by using some external api to attempt to GET something running on that port on the machine running the software. Voila, the port is open. The software can then request that the mapping it made be deleted. So can anyone else who can make requests to the IGD. The Software then effectively has a lease over this port, and the port will remain open until its lease expires, or the software deletes it manually. The maximum length of a lease is configurable by the IGD, but the max defined in the spec is 4294967296 seconds - a ridiculously high max.

The only other case where the lease will be terminated is when the IP that generated it is no longer around. If the machine shuts down or its DHCP lease over its own IP expires, the spec assumes the port mappings associated with it are lost too. This is not always the case though.

The RFC states that “we assume that the IGD applies the appropriate security policies to determine whether a Control Point has the rights to delete one or a set of mappings”. This is not an unreasonable assumption; after all, routers come in all shapes and sizes, and the authentication mechanisms they rely on come in various shapes and sizes too. It has become obvious with time though that due to the nature of the consumer hardware industry, security is an afterthought, if a thought at all. Indeed, a rather hilarious but sad security exploit reported by Akamai found that around 3.5 million routers scanned responded to UPnP probe packets. This means that external parties can punch holes through the firewall by sending UPnP requests to it. Even worse, around 277,000 of these routers actually don’t verify that the software making the request is port forwarding to itself! In fact, the RFC mandates this. If routers were to ensure that port mappings being requested by a source were to point at that source, they’d not be in compliant with RFC 6970!

This means that previously inaccessible hosts within a network can now be reached by attackers spamming the routers with UPnP port forward requests - effectively opening the machines behind the firewall to the open internet. Terrifying stuff.

It gets worse though. Some of these routers routers don’t do this origin check, and in addition they don’t even check that the port mapping points to a machine that exists on the local network. Essentially this means that any one of these routers can be hacked to become a sort of proxy. How? By asking it to, of course.

This is why I now disable UPNP everywhere I have the power to. It’s one of those convenient yet terrifying technologies I am not sure I can trust. Sometimes free convenience is far more expensive than anybody realizes. Just ask the NSA.

Sometimes, the degree of insanity in a conversation can reach such a fever pitch that it is hard to recover. Sometimes, people can say things that leave you so flabbergasted that you aren’t sure if you’re being trolled or not. This was one of those times.

It was another boring grind at AverageISP Tier 1 Technical support. I was getting very bored and was browsing Reddit as usual. Finally, my phone rings towards the end of the night.

I’ll just transcribe the call.

Me: ‘Good evening, you’re through to the AverageISP technical support team, my name is Kevin, how can I help?’

Caller: ‘Uhhh yes my uhhhh phone isn’t working’

Me: ‘Okay no problem sir, can I start by taking your telephone number?’

Caller: ‘uhhh…er……uhhh……….ummmmm….0161..I think….6549864’

This brought up an account on our systems, so far so good

Me: ‘Okay thank you for that information and could you please provide me with your full name?’

Caller: ‘Uhh….My name is uhh……..Sarah..no wait…David Dodgyman’

Slightly suspicious me thinks… Let’s check.

Me: ‘Can you provide me with your full address including postcode’

Caller: ‘uhh my address..give me a few minutes……………..’

Caller: <gives address but seems unsure and doesn’t know postcode>

Me: ‘Okay thank you sir and can I take your date of birth?’

Caller: ‘I don’t know my date of birth’

Alright enough is enough, company policy being what it is, time to terminate the call.

Me: ‘Then unfortunately sir I am unable to speak to you regarding this account but I will run a line test for you since you have reported a fault and I will book it to be fixed if it fails’

Caller: ‘THATS ABSOLUTELY F****** RIDICULOUS’

Me: ‘all I can do is apologize sir..’

Caller: ‘F****** HELL THESE F****** ROMANIANS COMING FROM ROMANIA TAKING OUR BRITISH JOBS’

…Wat? My brain pretty much just BSOD’d at this point

Me: ‘Sir…I’m Welsh’

Caller: ‘WHAT’S BEING WELSH GOT TO DO WITH ANYTHING?’

Me: ‘What’s being Romanian got to do with anything sir?’

Caller: ‘ARE YOU ACCUSING ME OF BEING A RACIST?’

Me: ‘No sir I am simply trying to establish how Romania is relevant to this discussion’

Caller: ‘WELL I BET THE DATA PROTECTION LAW WAS PUT IN PLACE TO PROTECT PEOPLE FROM THOSE F****** ROMANIANS’

Me: ‘Sir the data protection act was implemented around the turn of the century to protect your information-’

And was mainly written by the British, but that’s a different fight altogether kermittea.jpg.

Caller: ‘THATS F****** RIDICULOUS - I WANT TO SPEAK TO YOUR MANAGER’ Me: ‘Sir because you are not the account holder nor a customer I will not honor that request and I am now terminating this call, thanks for calling-’

Caller: ‘F*** YO..’

Automated survey: Thanks for calling AverageISP today, we’d like to ask you a few questions about your experience today…

The two approaches for doing this in a timely fashion I considered are as follows:

1. Write some script that makes use of pipelining in order to stream requests to the Redis instance without waiting for the response
2. Writing some Lua script which redis can execute itself, an amazing ability that I should learn to take advantage of more often! This approach would likely have been much more efficient, but there were time constraints on this and my knowledge of Lua is nada, so I decided to go with the script approach.

Life without pipelining

Without pipelining, when making many requests to a Redis, we are at the mercy of the latency between ourselves and the server in question, or are forced to do threading of some sort. Let’s look at some code. In this code, we want to get the ttl, or time to live for each key. If you’re unsure what a TTL is, the Redis docs do a good job of explaining it, but the long and short of it is that you can set an expiry on a key that Redis stores. The Redis TTL is the number of seconds before the key is deleted.

keys = ['apple', 'orange', 'banana', 'pineapple']
values = []
keys.each { |key|
    values << redis.ttl(key)
}

In this code, we are getting the TTL for 4 keys. Every time we call redis.ttl(), our code blocks. This is because we need to wait for Redis to respond to our request for the ttl. This is usually fine for small numbers of keys, but when you’ve got millions, and the Redis you’re talking to is not on your local machine, every single key takes at least the network latency between you and the Redis to retrieve.

In the above example, we are retrieving 4 keys TTLS. Lets assume that I, in London am talking to a Redis in the US somewhere. Let’s assume a minimum network latency of 100ms. This means, to retrieve these 4 keys, it will take at least 400ms. 40 keys will take at least 4000ms, and so on. You can see how this doesn’t scale.

Enter pipelining

Lets look at the code changes required to pipeline our requests.

keys = ['apple', 'orange', 'banana', 'pineapple']
values = []
redis.pipelined do
  keys.each{ |key|
    values << redis.ttl(key)
  }
end

That’s simpler than expected right? What is actually different here? Instead of sequentially doing the requests, we pipeline them. This means we send the requests without waiting on the responses. We send as many requests as our machine can, and the responses come in whenever they’re ready from the server. This means if we take the same latency figures as above, we can assume the time to retrieve all these TTLS is going to be around 100ms! for 40 keys, the latency will probably be a bit higher, but closer to 100ms than 400ms for sure! 400 keys? Much closer to 100ms than 4000ms!

Here’s a diagram I stole from Wikipedia that does a great job of illustrating the difference this makes to how the requests are made.

In this library, the redis.pipelined block will block until all the responses come back, and the values array will be populated with Redis::futures. You can wait on these in a separate thread and do work as soon as results come in, but if you don’t care about that and just want to start your work once you’ve gotten all your responses, you can just let the redis.pipelined block continue to block until all the responses have come back, and then call value() on each element in the values array. For example:

puts keys[0].value # returns the ttl, ie 3600

How does this work, though?

At a fairly high level, what happens is the server buffers these requests we’re sending to it in the order that we sent them. This is why we can depend on the order they’re returned in. It processes the requests as it normally does and returns them to our client as normal. There are some important caveats which might matter to your use case though:

• Pipelining does not make your requests atomic. If you’re making millions of requests for the TTLS of all your keys, then we can assume it’s going to take longer than a few seconds. This means that if keys you request expire in the time it takes for Redis to process all these requests, you’re not going to be able to retrieve them again. If you need atomicity, you’re going to need to do a transaction.
• The buffering of requests on the server takes up memory. If your requests each take a non-trivial amount of time to process by Redis, this buffer can grow very large quickly. If you’re making millions of queries, you might want to consider batching your requests. For example, you might pipeline 10000 calls to the TTL method, and once that lot succeeds pipeline another 10000 calls. This means your buffer will never exceed 10000 requests, and you’ve ensured you’re not going to OOM your Redis. The size of your batches will decide how many times you’ll need to pay the RTT penalty.
• Redis has lots of lovely methods which somewhat negate the need for pipelining. For example, if you’re just getting the values of keys, consider using mget instead of pipelining.

Conclusion

Redis pipelining allows us to make many requests without waiting for the responses or worrying about the order of the responses. It offers huge performance benefits if your service that talks to Redis is far away from the Redis instance. Regarding my use of it, even with this optimization, the majority of the time my script executed was spent on the machine running the script making the modifications to the keys. I am certain more performance could be gotten, but this was more than fast enough for my purposes - and was a very significant improvement.

Every now and then I’d come across one and the question would briefly pop back into my head, but it’s surprising how high the barrier to giving in to curiosity can be. My curiosity came back more intensely after my wife pointed one out and said that she’d pondered the question herself for a long time. She’d also seen them around, but once again hadn’t been curious enough to search for it.

Some furious searching resulted in my discovery of a Twitter account along with some rather confused discussions online regarding the sticker. I also noticed the domain whysuspectus.com being free to register.

I registered the domain, thinking that it was a shame for these stickers to be everywhere with a related domain being free to register. I rarely buy frivolous domains, but this one just stood out to me for some reason. I bought it on a whim. I didn’t really do anything interesting with it. I set up something cute for my wife and also made it point to my CV, because I wasn’t sure what else to do with it.

I forgot about it for a few months. In the back of my head, I wondered if the original owner of the domain would notice, or care, or whether they’d totally forgotten about these stickers altogether.

A few weeks ago, I got an email from someone who I will refer to as Steve (all details scrubbed at the request of the artist in question).

Hey Kevin,

My name is Steve, I’m a NaN year old artist from Country.

In 2011 I designed the Why Suspect Us logo

In January 2017 I created a website at www.whysuspectus.com to sell t-shirts and art for my brand using my why suspect us logo.

The shirts didn’t sell.

I dropped out of university, worked really, really hard, full time as a barista for a year and a half so that I could travel around Europe working hard, putting up stickers.

While traveling, the credit card attached to the domain name expired and I see that you’ve since registered www.whysuspectus.com

I’m a struggling young artist in 2019 and its painful, gut wrenching and heartbreaking to see someone else take control of the domain name.

I am appealing to you to pass me back the domain I have spent several years curating - I am happy to compensate you for costs of transferring and hope you can appreciate my anguish at the current situation.

When I saw this email, I was pretty surprised. I had assumed whoever had placed all the stickers had just forgotten all about it. I also assumed that even if they’d noticed, they’d do nothing but chuckle about the silly use of their old domain.

After laughing for a bit, I replied:

Hi Steve,

I had a feeling you might contact me! I’m Kevin. It’s nice to hear from you.

I bought this domain after I wondered what the hell these stickers were everywhere. It seemed a shame to let a domain with stickers all over Europe go to some domain squatter, so I purchased it and set up some silly stuff with it.

I’ll make you a deal! I will transfer the domain to you free of charge, in return for a few of those stickers. I’d also love an explanation as to what “why suspect us” means - if it has a meaning! The domain is good for at least the next 10 months. To do this, You’ll need to register a Namecheap account and provide me the email you used.

You should consider yourself very lucky! A domain squatter would have asked for hundreds if not thousands. I highly recommend renewing your domains if you intend to keep them - even if you’re not using them. I’ve lost domains to squatters and it sucks.

Almost immediately after I get a reply:

Hi Kevin,

Thank you so much! It really means alot to me. Very lucky you’re a good guy

I have made an account with namecheap using this email address: steve@some-email.com

And yes! Where would you like the stickers sent!

I also have a few custom WSU T-shirts, what size can i send you?

Thank you Kevin I really appreciate it Steve Stickers on their way!

Past this point I verified the persons ownership of the domain via a method I won’t talk about here and transferred Steve the domain.

A few weeks later, I received my shirt, as promised.

In this story, the original owner of https://WhySuspectUs.com/ got incredibly lucky that a domain squatter didn’t extort him. I have lost domains to squatters and it absolutely sucks. Domain squatting should be made illegal. In this case, the squatter happened to be a curious engineer who didn’t want to let an opportunity pass by.

This makes me wonder whether domains are ’too’ easy to purchase. The fact that within 10 minutes I was able to squat a persons previous business could have been incredibly harmful. Imagine if I’d used the domain for defamatory purposes!

This story has little point, but was an interesting interaction and answered the question I had all along. It’s a T-shirt brand! The artist in question seems to have decided to stop selling them now though, so unfortunately you can’t have one - however you can check his website, just check the sticker.

Recently, I’ve had a bunch of people who wouldn’t normally be interested in such things ping me about Initiative Q, asking what I thought of it. When people who aren’t in the tech circles I’m in start asking me about something like this, alarm bells start ringing, given that usually tech people tend to know about these things before they become popular, so I investigated.

Firstly, and I can’t shout this loud enough:

Q is not a cryptocurrency.

Q is a centralized coin. It is issued by a central authority, and right now isn’t even traded. This makes it presently completely useless; not that being able to trade it would bestow it much more usefulness.

Right now, the website is promising to give users who sign up today over 100,000 dollars. Of course, they mean in future money, once Q has gotten up and running and is worth anything at all. How the hell they’re estimating the future worth of a currently worthless token is anybodies guess.

Q does not solve the problem it claims to

Q claims that it has found a way to incentivize shop owners to install their payment infrastructure, but as far as I can tell, their strategy is to give a bunch of people worthless tokens, which will somehow encourage shop owners to pay to install hardware so that they can take these pointless coins. I somehow doubt it.

If the number of users was the barrier to a shops adoption of a new currency, then why haven’t we seen shops en masse buying Bitcoin or Etherium payment terminals? It wouldn’t be that hard really to build a payment terminal that can accept a large array of CryptoCoins - and they’ve all got the benefit of low/no fees (lol low fees - 2022) and no central authority! Bitcoin in particular can do practically instant transactions (lol instant transactions - 2022) using the lightning network these days too!

It smells a lot like a pyramid scheme

Their marketing literally begs you to get others to sign up, and if you get others to sign up then you get more money! It even in a hilariously self aware fashion explains that if not enough people sign up, the project will fail. Unfortunately for the authors of this nonsense, ’enough’, means ‘infinite’.

Get rich quick schemes still work

What is surprising in all this to me is this:

• Users are visiting the homepage of this website, and are seeing a dollar amount promised to them in the future, if they just give their email now!
• After signing up, they’re being told that they’ll get 10% of that amount in worthless tokens, but if they want another 40% worthless tokens, they had better sign a few of their friends up!
• Users then are spamming referral links all over the internet attempting to get others to sign up with their link.
• Users don’t see anything wrong with all this!

Does this seem like a reputable scheme? A vicious threat to those fat cats at VISA and Mastercard? If only you’d share your referral link one more time we could have defeated HSBC. Like my Facebook page to take down AMEX. Share this with 10 of your friends or Western Union will come into your house and mess up your pots and pans. I doubt they’re losing any sleep over this nonsense.

How about a ponzi scheme to collect a bunch of emails for people who are interested in futuristic Fintech to potentially exploit later? Seems a bit more likely to me, but hey, what do I know, I missed out on my Bitcoin millions.

To this effect, I want my television to switch on and over to BBC news in the morning. Unfortunately, Hisense do not provide a way to control the television remotely as far as I can tell, and nmapping it doesn’t give me anything too interesting. The only way I can think of to interact with the thing is via Infrared, so that’s what I’ll do.

So far, I’ve acquired the following hardware:

• China-Arduino Nano
• KY-022 Infrared IR Sensor Receiver Module
• KY-005 IR LED Module Transmitter

I hooked this all together in a sensible fashion.

Next up was to figure out how the heck IR transmission worked. It turned out to be a little more complicated than I had imagined. It turns out there’s a bunch of different standards for data transmission (surprise surprise), from a bunch of different manufacturers. As usual, nothing is quite that simple and my television was not listed as a supported standard with the library I was using which is why I purchased the receiver.

Hopefully, I could ’learn’ codes from the remote control and rebroadcast them at will using the transmitting IR led.

A preliminary test using the IRecord example showed this theory to mostly work! It turns out that this particular television (the Hisense H43N5300) makes use of a standard called RC6, developed by Philips.

The next steps for me will be to learn the button sequence required to turn the television on, flick to the news, and potentially slowly increase the volume until I turn it off somehow. After that will come the fun process of disassembling a cheap alarm clock and fitting it all inside there, ready to be triggered by my alarm.

They’ve long since taken the site down, I suspect in response, but you can still somewhat experience it thanks to the Wayback Machine.

You can also experience NotSoEducated, thanks to the Wayback Machine.

I did however somewhat break this trend with an impulsive purchase of the Huawei Watch 2, which at the time was one of the best (technically speaking) Wear OS smartwatches on the market. I thought “Hey, it’s a Google product, how bad can it be?!”. As I’d learn over the course of owning the damn thing, quite bad.

The first thing I noticed about the watch was just how poor the performance was. Considering the fact that the processor inside of it is roughly equivalent to a Nexus 4, the performance was appalling. Even just swiping through the various menus caused frame drops and stuttering. Still, I powered through, and attempted using the watch for what I thought it would be useful for; music.

One install of Google Play Music later, and I begin to wonder why I’d bothered. The app takes around 20-30 seconds to launch fully (I swear I’m not making this up!) and you’re booted into a very spartan interface that can charitably be described as functional. Then, once I hit play on a particular song, another 10-15 second pause, and…what’s this?! The music is coming out of my watch! I’ve got a set of headphones connected to my phone, yet the watch decides the best thing to do is blast said music out of its own speaker! What use case does this serve? I’ve never seen a single person listening to music literally through their watch, so why is this the default?!

I later learned the intention is to pair your Bluetooth headset up to the watch, so that it can play independent of the phone. My question is why would I want to do that? I’ve got 3000+ MAh in whatever smartphone I’m carrying, why not use some of that battery power instead?!

I later learned that in order to control the music on your phone, you first have to take the phone out, start playing music, and then you’re presented with some incredibly basic back/forward/play/pause type controls in a notification on the watch. None of the nice navigation through playlists or stations. Wonderful.

Even more hilariously broken is the Google Assistant. One day, I was working out in a gym and I decided that the music coming out of my headphones could do with being a little louder. I hold the Google Assistant button on the watch, and around 5 seconds later it chugs into life. I ask it to turn the volume up. What does it do? Google “turn the volume up”. Another time, I’m going to bed and I want to be woken up in the morning, so I hold the Assistant button, wait the customary 5 seconds for it to finish mining Bitcoin or whatever the watch is doing other than responding to my request for Assistance(tm). After the Assistant finally wakes up, I ask it to set an alarm for whatever-O’clock I wanted it set for. Wonderful, it set an alarm! I felt assisted. I then take the watch off, stick it on its charger and go to sleep (a necessary process, seeing as it lasts roughly a day, maybe 2). Unsurprisingly, the assistant played a trick on me again, I slept through the alarm, which was coming through the watch, which was away on its charger, rather than the phone, which I kept beside my bed, fully expecting the alarm to come from there.

Another fun aspect of Wear OS is the app development ecosystem. It’s Android! This sounds great if you’ve ever messed about with Android development in the past, and in some ways this is nice, but frankly it feels incredibly bloated and wasteful considering the majority of the apps on the watch are garbage anyway.

The common theme with all of these problems is this insistence on the watch being independent from the phone, and I just don’t understand it. Surely, it would be better to treat the watch as an alternative method of interacting with the phone, where it makes sense for this to be the case? Why, when I ask the Google Assistant a question, does it deal with the query itself? Why not forward it to the Google Assistant that is running on the phone itself? There’s only so many times you can ask the assistant how tall the Eiffel Tower is. Why, when I have headphones plugged into my phone, does the watch assume the best audio output available for music is the watches speaker?

It’s not all terrible though. The Citymapper app for Wear OS was a glimmer of hope. In the morning, my phone pops up telling me that it’s calculated my route to work for that day. I tap the notification, and immediately my watch vibrates and starts displaying the route information. It sounds ridiculous, but that kind of interaction is extremely rare.

Comparing and contrasting the Citymapper app to the Google Maps app, it’s almost comical. Remember that these devices have screens that aren’t much larger than a two pence piece. Opening the Citymapper app (which launches pretty much immediately) on the Android Wear device offers you two options, “Get me to work” and “Get me home”. Anything more than that, you’re going to need to pull your phone out. Open Google Maps however and you’ll be waiting approximately 7-8 seconds for it to literally load a street level map onto your wrist. Dragging the map around stutters and skips frames. Can you get directions to your home or work through this App? Of course you can’t! You can, however, search for places nearby, and get navigation directions to those places, if they’re in the list of Google approved(tm) places. Why, in Gods name, would anyone want to look at a two pence piece sized version of the world? If I wanted to see a map, I’d have looked at my phone!

I think that Wear OS would be so much better off if instead of working to put a phone on your wrist, it was instead designed to be auxiliary to the ridiculously powerful slab of glass you already almost certainly have in your pocket. While I don’t regret the purchase, and to Googles credit things do seem to be very slowly improving, I certainly won’t be buying another Wear OS device. Companies like Xiaomi are releasing devices like the Amazfit, which while nowhere near as versatile as Wear OS, give me the main things I care about: Seeing and responding to notifications, tracking fitness/sleep, controlling music, and telling the damn time. It also has a month of battery life, and isn’t trying to do literally everything the phone in my pocket does better than it.

Update: I eventually sold the damn thing and have been much happier since.

The device was an interesting one, and to describe just how far ahead of its time it really was, it will be useful to draw comparisons against smartphones of today. Like a smartphone, it supported instant messaging, had a collection of downloadable apps that received updates, allowed those apps to communicate with one another (for multiplayer), had a SDK to allow others to develop apps in what they called “CyBasic”, and even received operating system updates! Bearing in mind this was released in 2001, when the most popular phone was likely Nokia 3310 vintage. It even did some interesting stuff around mesh networking that never really worked, due to the relative unpopularity of the Cybiko and the short range of the wireless.

Where this device differs from your average phone though, was its lack of GSM support. Hilariously though, if you purchased a wireless adaptor that would plug into your PC, your Cybiko then could browse the internet via WAP. I wonder how many other devices that didn’t have GSM supported WAP?

The device was loaded with lots of forward thinking functionality. It had an expansion slot into which one could plug extra devices. One such device was an MP3 player, which took compact flash cards, and then allowed apps on the Cybiko to interact with it.

I think its downfall was its extreme focus on releasing hundreds of games for free, rather than releasing a few good quality games and charging for them. It was competing against the already 2 year old Gameboy Color with its monochrome low resolution (and fairly terrible) display. It also had fairly terrible audio too. Whereas the Gameboy was laser focused on playing games, and as such had the controls to match, the Cybiko came with a stylus. The stylus was not for a touch screen however, it was for pressing the ridiculously small keys. As usual, content is king, and the Cybiko game catalogue just didn’t have the content to really ‘sell’ it as a platform.

They did release a follow up product called the Cybiko Xtreme, which featured a much improved operating system, a higher contrast display, generally faster hardware, a much slimmer profile, and a significantly better keyboard, but it was too little too late. The Xtreme failed to address the main problems Cybiko had - the poor quality game catalogue and the low penetration of the device rendering the interesting wireless functionality useless. The Classic unit could be found in retailers for as little as £10, towards the end of its life, along with whatever remained of the Xtreme stock for £30. A far cry from the original asking price for both.

I owned and loved both as a kid, and fondly remember them. Rest in peace, Cybiko!

Assets

A player controller that:

• Allows for forwards, backwards, and rotation motion
• Keeps the player locked to the surface of any given geometry, provided that it has a mesh collider (Think Super Mario Galaxy)
• Is compatible with VR headsets - tested with Oculus Home and Rift DK2.

A Virtual Reality User Interface controller script that:

• Implements a gaze pointer type interface, where the user looks at three dimensional objects in order to select them
• Follows Googles’ guidelines on VR interfaces
• Describes an interface (IObjectAction) that can be implemented in scripts attached to 3D objects either generated during development or pragmatically, to allow for totally new interfaces
• Implements a non-blocking timer that is used to decide if the user has been looking at an element long enough
• Displays said timer in three dimensional space to the user wearing the headset.

Quickstart

To get started

1. Clone this repository
2. Open the ‘World Explorer’ folder within the repo as a project with Unity.
3. Hit Play in Unity and use your standard method of control! Feel free to build on this scene.

Implementation details

Creating a new scene

1. Create a Capsule, and give it a Rigidbody, disabling Gravity. Add a Camera as a child of this Capsule and ensure its’ position is set sensibly; (0,0,0) is usually a good bet.
2. Attach the genericWalker script provided to this Capsule.
3. If user interface is desired, Firstly attach CameraLookAtUI to the player camera.
4. Tag the game object you wish to do something if the user looks at it with either “Major” or “Minor”. This distinction allows you to set two separate activation delays for these two different classes of object.
5. To define the action you want to take place when the game object is looked at, Create a new C# script called whatever you like, and implement IObjectAction, concretely defining activate() - where your action takes place.

Example

    Example which selects a different scene:

    using UnityEngine;
    using System.Collections;
    using UnityEngine.SceneManagement;

    public class goHome : MonoBehaviour,IObjectAction
    {
	// Use this for initialization
	void Start () {
    }

	// Update is called once per frame
	void Update () {

    }
    public void activate()
    {
        SceneManager.LoadScene(0);
    }

Licence

    Copyright (c) 2016 Kevin Norman.

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the "Software"),
    to deal in the Software without restriction, including without limitation
    the rights to use, copy, modify, merge, publish, distribute, sublicense,
    and/or sell copies of the Software, and to permit persons to whom the Software
    is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
    OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
    IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
    TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
    OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Github

https://github.com/kn100/worldExplorer