It was hard to understand their purpose in the grand scheme of things though. Sometimes they were buried deep inside multiple layers of function calls.
I must say this is often true even when I'm looking at the source code. I think the vast majority of software out there is heavily overabstracted, and MCUs have gotten big and cheap enough (two ARM cores in an HDMI capture device!?) that this is appearing in embedded stuff too. This device shouldn't have needed more than one MCU.
Growing up in India about 20 years ago, we often repaired or renewed almost everything because our buying power was low and things were expensive. We used a lot of hacks, known as *jugaads*, to make things work. Even clothes were reused, with tailors doing *rafu* (patchwork) to extend their life. This was especially common in middle-class homes like mine.
My dad, who worked in a garment export house, used to tell me stories about how people in the West preferred disposable items and often opted for newer stuff, whether it was cars, gadgets, or clothes. At the time, I didn't understand this mentality. But now, with increased buying power and lower costs (thanks to China), we too tend to just chuck things away and get replacements.
I deeply admire people who don't give up midway and think it's easier to buy new. This type of persistence and resourcefulness is truly commendable.
Cost to repair has grown while cost for new has reduced.
If you can repair things yourself, and you can find the parts you need to repair things, then sure. But if it's something I've got to pay for someone's experience and wisdom, that's pretty expensive these days, at least where I'm living, and it's just plain hard to find people who repair things too; lots of signs for TV repair outside empty shops.
Thankfully I'm semi-retired, and am on a salary for part time work, so my marginal time has no dollar cost, so I can take a day to try to repair a dishwasher, and then another day to install a new one when it rebreaks a couple days later. A professional installer probably would have had the new one installed in an hour instead of my all day, but I didn't have to wait for scheduling, at least.
This was how things went for a long time in my region (western europe) as well, my parents grew up patching clothes and repairing stuff a lot. It's only in the past 50 years or so that consumerism has gone up and the quality and cost of e.g. clothing has gone down.
I've been doing maintenance on my motorcycle myself recently, it does take some small investments in some tools to get started (like a tool to undo the oil filter, although in hindsight a strap and a stick would do the job) and you need to source some parts and replacements (fluids, copper washers, but also replacement screws for the weathered brake fluid reservoir ones), but it's in the region of €100-€150 instead of the €1000 the garage quoted me for.
> It's only in the past 50 years or so that consumerism has gone up and the quality and cost of e.g. clothing has gone down.
The “quality” part is a big factor, cost optimisations and fast turnaround means it’s often not worth repairing things at all e.g. a fast fashion T designed to survive for a season (if it survives even a wash).
An other major issue is scams around price signals and brand degradation. It used to be you got what you paid for and some brands were known for quality, so you could pay a fair amount of money to a reputable brand and you’d get stuff worth maintaining and repairing.
But big groups and P-E have taken to “value extract” from brands, so they take a reputable brand and start white-labelling / cost-optimising, initially keeping prices in order to get maximum money for the moo their start selling instead of milk. Then they drop the price as understanding slowly spreads, until a once reputable brand becomes bargain-bin fare even to the general public.
There’s a similar issue around more bespoke products, which optimise for quality signals (e.g. external design and materials) and sell generic inner parts (or outright garbage) for top-shelf prices.
Then there’s the shuffling of 6 months brands on generic white label goods (amazon is absolutely infested with that, you’ll get the exact same product under half a dozen brands, and 6 months later most of those have disappeared).
It seems to me the real problem is that repairs are, in general, labour intensive. Few products today are sufficiently valuable that a repair is better than a new item.
Cars are worth fixing, a 10$ shirt is not, if you value your time. This only becomes more true as expertise becomes required to effect a repair, since you become less capable of repairing and the time of the repairer becomes more valuable.
Yes Reflecting on it, making things last longer had some great side effects. For instance, almost every woman in my family knew how to *rafu* clothes (1), and people understood how things worked under the hood of a car (like you my father did all the maintenance too). These skills were passed down through generations, becoming a part of our everyday knowledge.
I guess a lot of things aren't that simple or accessible as most of it is often a black box nowadays. But anyway, Skills like these not only saved money but also fostered a sense of self-reliance, resourcefulness and stuff your parents taught you as life skills.
The best tool I have bought in the last 3 years was a 3d-printer... It lets me make other tools - even if they aren't as durable as steel, I can design them chunkier, or print a new one if they break.
You can avoid the copper washers! The main reason to change them is under compression the copper work hardens to seal up. To get a good seal though you just need to re-anneal the copper so it's soft - heat it to cherry red and let it cool down. Takes about 5-10 seconds with a blowtorch - I've been doing it for oil changes on my car for several years now with no problems.
This is how it was for me growing up blue collar in the northeastern USA in the 80s. My father fixed everything in the house and the vehicles. I inherited my older siblings clothes, and my younger siblings inherited mine. My mother would hem pant legs shorter when we were young, and then let them back out as we grew older. If you wore a knee or an elbow out of clothes, it was getting patched.
This instilled some good and bad tendencies in me. I do almost all of the repairs around the house myself. I work too much though, so I don't always have enough time or energy. Even though I can easily afford it, I have a hard time paying someone else to do them. This means I live with broken stuff longer than I should.
I'd probably have more money if I spent that time working on side projects instead of doing maintenance and repairs.
> This is how it was for me growing up blue collar in the northeastern USA in the 80s. My father fixed everything in the house and the vehicles. I inherited my older siblings clothes, and my younger siblings inherited mine. My mother would hem pant legs shorter when we were young, and then let them back out as we grew older. If you wore a knee or an elbow out of clothes, it was getting patched.
Thing is, they were able to in the first place.
Forget about fixing a modern car. The electronics side is a mixture of "a datacenter on wheels", DRM and anti-tamper technology (sometimes enforced or heavily suggested by law such as in emissions control, sometimes by reality, e.g. "Kia Boys") and high-speed protocols instead of early age wires and relays that you could troubleshoot with a decent multimeter. The physical side is a ton of plastics designed to absorb crash energy and finely tuned metal alloy stuff (with the form also having crash safety implication) that your average DIY person cannot reasonably weld instead of plain old steel sheets. You can't buy a "reasonably repairable" new car any more because of the legal mandates and because you don't want it to be stolen by some kid having watched a YouTube or Tiktok video showing how to bypass the locks.
And clothing... patching a 1980s piece was possible, the fabrics had weight and structural integrity of their own. Nowadays it's extremely thin fabric everywhere that shreds itself after a few washing machine cycles. Try to patch it and you'll more likely than not find out that your very act of pushing a needle through it to apply the patch just causes the next rip to appear. You are still able to purchase better quality clothing technically but you end up paying like 4x the amount and it's still made in some Bangladeshi or Chinese sweatshop under horrible safety and employee rights standards.
If one has the proclivity, then: One can get into rather far into troubleshooting and (and ultimately repairing) common modern automotive electronics with an Autel rig that, adjusted for inflation, costs less than an Atari 2600 did.
I feel like it’s giving good money away when you hire someone to do work for you that you know you can do. You look at the markups for things and it gives you pause. Things like a valve or whatever. You can go to the Home Depot and get it for cheaper even when you include the cost of whatever tools you need to get.
But at some point you have to say, let’s just get someone to to it (the deck, the fence, the gutters, etc.) still it’s like zen and the art of motorcycle maintenance. There is some personal satisfaction in being self reliant.
In our past, this seems a community thing: Someone in your neighbourhood had advanced knowledge in welding, someone else advanced electricity, masonwork, clothes repair, ... They could coach other people to a basic enough level and help if unexpected troubles popped up. The community as a whole had knowledge and basic apprenticeships built in.
Today, repair is something you do on your own. Things like youtube are a great help, but the community aspect is lost.
When I bought my phone, it was on sale and so I got it new for $94. I dropped it and broke the screen. The replacement screen cost $106, more than the entire phone (just for the part, I do the work myself). I still did it anyway, for environmental/e-waste reasons and because repairing electronics is a fun hobby but the sad reality is, it often makes more financial sense to replace than to repair. That's not even going into the time lost and the money sunk on tools.
I grew up lower-middle class in America and to and extent I understand this.
Disposability is especially offensive when it comes to ~computers.
To have perfectly functional (from a hardware perspective) 10> year old smartphones become "e-waste" is absurd to me.
Even a cheap smartphone is a remarkable achievement in manufacturing and engineering. Its wild to think of something like that as "junk" even though it effectively is.
The quality of workmanship and repair has gone down drastically in India now. People have figured out how to make money by keeping things locked. Repair is just swapping out parts and cleaning or go for the next model. The mentality of people has also changed drastically with them changing products in short years. Consumerism has gone through the roof and there is an active disdain of people who preserve and get the product to function for a long time.
My mom repurposed old pants as bags (I used it to carry my books at college - did get a lot of odd looks, but that's what I could afford at that time). Even now, I cut pieces of old clothing to be used for cleaning purposes. Those habits die hard.
This was my attitude towards cars when younger. Always bought cheap used ones and fixed them. It frustrated me to no end seeing people fail at basic maintenance and cut the lives of their vehicles by half or more compared to a maintained one. (Stuff like not changing fluids or driving with the fluids leaking out, driving with slipping belts or broken suspensions, etc.)
This is fascinating. How were these usually jugaads obtained - did you more commonly get them from friends, family, shop owners, the Internet, or just (re)discover most of them yourself?
I agree that repairing is vastly preferable - I grew up in a poor communist country, where repairing and self-resourcesfullness was the norm.
However it's not the consumerist mentality driven by increased buying power - it's sheer economic sense - the time and effort is way better spent, economically speaking - by simply replacing. It's the lower price of goods that drive this thing.
My fridge broke the other week. I called the repairman who quoted a repair bill that was 10% MORE EXPENSIVE then buying a new fridge. Simply the time of the repair shop and the transportation from home to shop and back was more expensive them just buying a new fridge, chucking out the older one, and calling it a day.
My grandparents, nah, my parents would be horrified by throwing up a "nearly" good fridge. To me, it makes economic sense.
This is often due to the total costs being externalised (pushed off to others) and therefore not reflecting the true cost of the replacement nor the costs of (safe) disposal of the old unit.
Externalised costs such as emissions from manufacturing of new raw materials (metals, plastics, gases, etc.), transportation, disposal, and more.
Obviously it depends on what exactly fails. I've kept 'white goods' going for over 20 years despite:
1) known defect where Hotpoint Fridge/Freezer evaporator thermistor fails due to freeze/defrost thermal cycle. Replaced more than 10 times; cost of new thermistor is pennies; time to replace (after initial explore) 10 minutes.
2) Freezer control PCB misreading thermistor; replace PCB: UK£35.
3) LG Washing machine bearing failures; replaced about 6 times; time to replace (after initial explore): 45 minutes.
I think sometimes repair-or-replace depends on one's state of mind. Figuring out what is wrong and how to fix can be frustrating but, equally, it can be extremely satisfying to realise you can do it and are no longer reliant on some mystical "expert" !
Society as a whole in many countries is losing (or has already lost) the ability to be self-reliant and that lack makes people and communities generally more fragile.
Self-reliance is one of the drivers of hackers and tinkerers.
My economic theory is that the price reflects ecological costs fairly well.
At first glance it might appear that fixing would have a lower environmental cost. But the money spent will be spent by the repairman on things like international travel or whatever - and each of the things the money is spent on have environmental costs and externalities.
I don’t quite think so. I think we’re in a status quo that prevents/obfuscates a more efficient economic activity because it inflates GDP which makes the politicians and economists happy.
The magic number won’t go up much if you call somebody and they tell you what you need to do change the PCB and ship you the part(for a small markup price).
I often develop feelings for the products I use (I know...). When I look at dishwasher I reminisce how many moments I had whilst standing next to it tirelessly working through my dirty dishes. I'll give it a tap. Sometimes I talk to it when loading like "Hey there, I got you some new stuff. Don't worry I'll feed you salt at the end of the week. Now I'll do your favourite program". Then once it finishes I say like "Oh what a great work you did there!" and so on.
Then when it broke (the motor seized) I just wouldn't have heart to simply dispose of it. I sourced the motor and called in repair guy who installed it. It did cost me in total as much as I would pay for a new dishwasher, but I would never get the sense of feeling that I saved a friend.
I am so glad that I'm not the only one. Every time I shut the hood of a car after working on it, I give it a little loving pat-pat.
I have this inexplicable feeling, contrary to my usually rational self, that machines have a sort of soul and "feel better" when they're taken care of, and I feel like I'm letting it down when I extend an oil change/put off maintenance/don't take care of a problem I'm aware of yet. I don't really believe they have souls or anything; it's just a feeling I get.
Come to think of it, I do the exact same things with my plants too.
I live in Sweden, coming from the Netherlands, both countries somewhere in the global top yet I still repair hardware, mend my clothes, repair the tractors and car, restored the 17th century farm we live on and extended it, built a barn and more. I intentionally do not make myself lust after the 'latest and greatest' of anything since I realise that such a lifestyle puts you on a treadmill, always running for the next treat. Hence I'm typing this on a computer from 2009 I got for free because the video card was 'broken' (27" iMac, a short stay in the oven later fixed the video card) connected to a second monitor I got for free because of a broken power supply (two capacitors later it worked again) which sits on a stand-up desk I got for free because of some trivial electrical defect (quickly fixed). In a way I still partake of the 'fruits' of that latest-and-greatest lifestyle, only with a decade or so of delay and without the compulsion to 'upgrade'.
Why do I do this? For a few reasons, most of them quite basic. I like fixing things. I get far more satisfaction out of using abandoned hardware which I have fixed myself than I get out of using whatever new gizmo I happen to lay my hands on because I know I can keep the former working (or find an alternative which I can get to work) while I do not know that for the latter. I like being self-reliant. With a soldering iron, a BGA rework station, a few old oscilloscopes and meters and a few decades worth of experience and scavenged parts I can keep things working for the most, design and build circuits to extend whatever is needed, etc. The advent of cheap and relatively open microcontrollers - the ESP series, Arduino, Raspberry Pi pico etc - has given a boost to the DIY electronics sphere which adds to the appeal of keeping older stuff working, e.g. I'm currently looking in to replacing the worn out control circuit and assorted switches of our 35 year old oven with something totally different and more functional, not because I can't get a new oven but because the current one works quite well apart from those switches. The same goes for the tractors and car, motorbikes (Russian Ural and Ukrainian Dneprs with sidecars), etc. There are no electronics in my tractors, they are purely mechanical. They can be repaired by anyone who knows how without the need for proprietary tools, dealer-only computer terminals and such.
Of course I could save a lot of time if I abandoned this 'life style' and just went with the flow, buying new clothes as soon as the old ones needed mending, buying a new computer every 3 years, a new car every 5 years, a new tractor every 10 years, a new dishwasher every 7-12 years, a new washing machine every 10 years, etcetera. I could stop cycling to the village and just take the motorbike, that would be much quicker after all. Think what I could do with all that time saved:
- instead of cycling to the village I could spend time at a sports school for exercise
- instead of fixing that computer (and learning a bit more every time I fix one) I could watch some series on some streaming service
- instead of mending that hole in my trousers (using the Elna sewing machine I got for free because it was 'jammed', took me all of 5 minutes to unjam it) I could browse the web looking for some new trousers - something I'd have to do every few weeks since my clothes somehow seem to acquire holes quite easily, why would that be?
- instead of building that barn I could be working a few more months to pay someone else to build me a barn
- instead of gaining self-reliance I could make myself become more and more dependent on outside sources and 'experts'
Well, thanks but no thanks, I'll just keep on mending my own stuff simply because I can and I like it that way. Here, in Sweden, in the land of plenty.
If you're interested in extending the service life of that old iMac, I recommend OpenCore Legacy Patcher. It lets you run newer versions of macOS on old hardware.
The "consumerism" mindset is a luxury created by a strong economy and high upward mobility. An unfortunate side effect is that as repair & reuse became less desireable, so too did repairability and logevity as features. Now that economic growth in the US has stagnated (particularly for the 99%), it's becoming aparent what we lost as nothing is made to last and companies are unwilling or unable to offer repairable, long lasting products. So even those who go out of their way to repair their property encounter a myriad of roadblocks.
>A light bulb went off in my head. Yes, these chunks of data in the flash chip were pictures! But not in the conventional sense. They were describing the various LED animations. Each set of 16 bytes is an animation frame written to the 14 LEDs. In the example above, a single red LED turns on and six of the seven white LEDs do stuff. The last two bytes in each row are a delay time before moving onto the next frame. The first 16 bytes looked kind of weird though. They looked like they possibly contained some header data for the animation or something.
I have independently invented this format, albeit in text instead of binary, for animating my Hue lamps. Animations in my version look like this:
bulb1 bulb2
dly R G B WW CW R G B WW CW
100 10 25 20 0 0 25 10 20 0 0
200 10 50 20 0 0 50 10 20 0 0
where the first value is the number of milliseconds to delay before moving to the next line, then each led channel's brightness (out of 100) is represented on each line. A line that starts with anything but a number is a comment, and any number of spaces (1+) are allowed between channels.
I'm tickled to see a similar (binary) format in a commercial product. It really is quite obvious, but still interesting to see.
I suspect it may have taken the author weeks as well, given they indicated waiting for chips from China and that there was communications with Elgato's customer support. This writeup looks like it's the report of a few weeks if not months of on and off work.
I'm the author of this article and you're absolutely correct! This was a long, drawn-out project. For some context, I ordered the replacement regulators in February. The new LED driver chips were ordered in March, so that was around the time that I actually had the failed hardware fixed. Then everything sat idle for months. The firmware reverse engineering to figure out the LEDs was several weeks of on and off work in my spare time.
This type of thing is definitely not something you can just figure out in a couple hours (or even days).
I most appreciate the honesty about mistakes and failures and dead-ends. Too often wasted effort and small failures are elided from articles and instead you get something that is written retrospectively as though everything went perfectly...
How do people maintain motivation on projects like this? Repeatedly having to order parts and wating for them to arrive, especially the one from China, make me abandoned an unfortunate amount of projects.
The time, and also the cost of ordering these parts. I know each is like a few cents, but you don't know all the parts you will need beforehand. So if I counted correctly he ordered at least 3 times.
Same here. I can totally imagine the excitement in their engineers eyes after reading this fascinating writeup, paired with their managers anger while they ask for a better way to protect the company's IP.
It’ll have to come from Corsair who now owns Elgato. IMHO, they’re a ‘conventional’ company with stock listed on the exchange. If they respond I will be pleasantly surprised.
Buy one share of their stock, then call investor relations and ask them why a hobbyist was able to fix the LED bug that their support claimed didn't exist while their whole expensive engineering team was not.
>while their whole expensive engineering team was not
Question is maybe not if they could, but if they were allowed to. If the project is already out of development and a bug is detected, sometimes there's only a barebones team left that barely gets the time to do basic maintenance.
You don't know what kind of pressures they are under to deliver.
I know of significant compromises in all the software I have written, and I would make hardware mistakes if I had taken the electronic engineer path. Sometimes we have to prioritize and I've never had an unlimited time budget on any project (not even my own).
A poor engineering culture usually exists because management broke it, rather than because the engineers failed to create it. Of course I have no insight into how Elgato is run, so I am not pointing fingers. I merely state that there is certainly evidence that their engineers are not focused on quality.
Well high pressure to deliver is sort of a poor culture isn't it? I agree we should be forgiving of others mistakes, but I also think we ought to strive to develop better solutions that lead to fewer bugs at any given velocity.
It's a culture issue when such cases don't get caught in testing and support staff either doesn't know about it (=the scripts are bad), doesn't get told that there is a workaround (because clearly there is), or (the worst of the possible options) gets told to act like everything is fine.
It's a race condition which involves physical interaction with the product, which only occurs during a very rare operation, which looks pretty much identical to a genuine hardware defect.
This isn't something you can just trivially unit test. If you don't see this happen multiple times during initial hardware development, you are never catching it. A single failure of a prototype can easily be attributed to a manufacturing defect - especially if it was hand-soldered.
Once it's in the field replacing the 0.01% of units suffering from random issues under warranty is far cheaper than having an engineer spend weeks trying to diagnose every single weird failure mode. Unless it affects a number of units, it's just not worth the money. You have to consider that support doesn't get a "the LEDs stopped working when I unplugged the device immediately after flashing it" message, they just get "the LEDs don't work". Support scripts are made for horses, not zebras.
> Once it's in the field replacing the 0.01% of units suffering from random issues under warranty is far cheaper than having an engineer spend weeks trying to diagnose every single weird failure mode. Unless it affects a number of units, it's just not worth the money.
It does affect a number of units, that's the point, and it's serious enough that the brand image is suffering. Just google "elgato unreliable" - tons of results, and when the top result is a Reddit post in the official Elgato subreddit of all places literally titled "My elgato experience has left me with nothing but hate in my heart", all alarm sirens should go fucking off.
To top it off, Elgato used to be a German brand right out of Munich [1]. German products used to be noteworthy for top-notch engineering and reliability...
It’s sad because their stuff, like the key lights and the stream deck are neat and really useful. and the software has all the features you’d want and you can make your own plugins for it and they document the process for making the plugins. All really cool.
But in practice I’m constantly having to power cycle my key lights and having to restart the stream deck app on windows.
I maintain a lot of their stuff for my coworkers. It's nice looking, and easy to understand, but they also have a lot of things that are kludgy clunky or strange that don't make any sense.
Their ring light, for instance, can't be manually turned off using the shoulder buttons. It can only be set to a low dim. Why not? Makes no sense.
Their FaceCam needs a high bandwidth usb 3.0 port all to itself. Won't reliably work over a hub. Why not? I mean, give it a low bandwith 720p mode or something.
The Mic has a headphone jack, sure, no problem, but why does it install 8 audio devices during the install? Only install what is needed!
Their USB capture cards have issues staying connected unless they are plugged directly into the laptop/desktop motherboard, and even then they flip out every few hours.
These issues have ensured that I won't buy any of their stuff. I've recreated an entire elgato stack that would have cost $500+ for like $90 in cheap chinesium crap and it works perfectly.
If you don't have a thermal camera, you can also paint rubbing alcohol onto the board and watch where it evaporates the quickest. Obviously, this only works for low voltage stuff...
Interestingly I discovered that if the chip is set for unprotected mode, the bootloader actually waits around for update commands instead of booting the application firmware. So Elgato is using the locked/unlocked bit as a way of signifying whether it should stay in the bootloader or not, which I find to be a little bit weird. That completely explained why my second HD60 S stopped working after I installed my unlocked firmware — it was stuck in the bootloader waiting forever because the chip was unprotected.
Great work! I find it bizarre that they even bothered with this wild SPI Flash muxing thing if all it does is drive some LEDs. Seems massively overengineered.
The ability to explore & chase down truths is what makes us divine, is the aspect of us that builds & enhances. This guy walked a deep deep spiritual path to exploring something of the mundane world.
It's just so sad, so frustrating, so hellish & infernal that devices do resist us. Computers & devices should be amplifying & radiating the truths they are made of. But humanity is trapped in shells of their own making, in descending consumerism that rejects engagement & learn ability. Computers, alas, especially so, are often the prime obfuscater of truth and understanding! What a vast disuse of so much potential!
I hope we see some break through currents, of machines that are open, emerge in my lifetime. Well done, this dude Doug.
If the right to repair movement succeeds fully, OP would have had access to the complete source code and hardware schematics of this device, and the fix for his flashing lights probably would have been a 5 line shell script that someone else had already put on github.
> would have had access to the complete source code and hardware schematics
Forcing every company to open source their products is a pipe dream. This is never going to happen.
Even if one country passed such a law, manufacturers would move their production to a different country. The products would then be imported and redistributed as foreign products, circumventing the law.
People would have said the same 100 years ago about forcing food companies to reveal their ingredients list. Yet now it is required in almost every country, and doesn't appear to have been that detrimental to the food industry.
Hardware schematics and full source code doesn't tell you everything about how a piece of hardware is made either.
How was the hardware tested? How was it calibrated? How was it assembled?
An ingredients list tells 90% of the story about how food is made, and in most cases an expert could guess the remaining 10% to get a decent result. Likewise, hardware design+code tells 90% and an expert could figure out the remaining 10%.
The government could totally require all consumer products have published source code. (published source code != a license for others to use the code)
To what level would the "source code" have to be published? Chip-level? Should I expect HDL code that allows me to reproduce the microcontroller?
If not, expect a bunch of gadget companies to pay cypress or whoever to make "custom" chips with their firmware burned in. After all, what's the difference at that point between HDL and firmware?
If yes, expect most companies to simply refuse, and not authorize their chips for sale in the US. International IP / licensing agreement would make it literally impossible for them to comply without being sued into oblivion by the Taiwanese company they licensed the IP from.
I think stuff like design files and source code should be held in escrow by the government. If you continue to provide replacement parts to customers, the information remains protected. Once you stop providing reasonably priced replacements ("reasonably priced" = less than the cost of the product), the information gets published so others can reproduce it.
We need to encourage our kids to repair stuff. It aids the environment (less e-waste) and teaches them a valuable trade. I know that's it's typically cheaper to replace than repair, but to me that's a market failure.
Industrialization has massively optimized the "hot path" for any kind of production, largely by taking out the humans. It is incredibly cheap to manufacture consumer electronics - because everything is standardized and predictable and amenable to machine operation. This is what market success looks like.
The downside is that as soon as there is a deviation from the process, it gets more expensive.
It's worth thinking about what happens to repair inside the factory. Some percentage of units will be coming off the production line defective and fail initial QA. This is usually a very low percentage, well below 1%, because the entire economics of the factory depends on not having to do any rework. Even there, the technicians will take a look, determine if it's something that can be fixed quickly, or is a novel kind of failure, and if not just throw it away right there. Why? Because while you're standing there looking at it, another hundred have come off the production line. The broken one in your hand is not a unique snowflake.
At the start of a run though, this is an interesting and important job, because any failure is a novel failure. The first batch through the process should have high yield, but it might not, and then you stop the line ( https://mag.toyota.co.uk/andon-toyota-production-system/ ) and figure out what's happened (something misaligned? Defective inputs? Material problems? Design issue?)
The first batch will often get reworked and sent out as demo units.
As other commenter says, spending human time to save a $15 chip is uneconomic. This only happens if they're especially expensive parts or (as during COVID) supply is short and you can't just buy some more.
(COVID shortages saw the opposite phenomenon in a few places: working consumer electronics being bought and stripped for a particular part critical to something else, then the rest thrown away!)
Remember that after you've hotgunned it off the board and cleaned the solder you have to re-ball BGA parts. Again, a process that's cheap in the original manufacturing line and very hard to do by hand. It also means the part has been through more thermal stress which will shorten its life. You don't want to have to rework a unit again if you put a recycled chip in it which fails.
The engineering time to "salvage" and test the 15$ chip, and the cost of downtime reintegrating the part to the production line is probably worth more that 15$, so unless the lack of that specific part is a bottleneck, probably no?
Yeah, probably, but I do wonder where it starts to make sense to desolder a chip and give it another try.
Surely at the point of high end GPU Boards one would invest considerable amounts of time and money to salvage a chip, which could sell for thousands of dollars.
I think you're right that for high end chips, it is worth the time to salvage it[1]. But I think that's the vast minority, like you said, most stuff being manufactured isn't at the cutting edge of consumer tech like GPUs are.
[1] They show a bit of the RMA/failed unit process towards the end of this surprisingly good GPU factory tour from Linus Tech Tips. There's some discussion of de-soldering and testing retrieved components. https://www.youtube.com/watch?v=GS35VHEfFDU
Not the user you replied to from my experience usually it's not worth it due to several reasons:
- desoldering the chips takes time and is a manual process ($), with risk of tearing off a pad or bending leads. In case of BGA ICs reballing is needed to reuse them.
- components are usually not rated for a lot of reflow (heat/cool) cycles, and some are moisture-sensitive and may crack if they have managed to absorb moisture
- you usually end up with some solder and flux left on the IC, which can cause issues
- ICs come on tape for feeding into automated pick-and-place machines, so you would need to feed and mount them manually ($)
And if you only realize you have damaged the IC after mounting it on the new board you end up having to rework it again ($).
Sure, it might be worth it if the chip is really expensive or hard to get, or you're soldering everything by hand anyway, but usually the math just doesn't work out.
Most devices nowadays are unfortunately not only not repair-friendly, they are straight impossible to take disassemble. They are welded, soldered, glued together, and any attempt at a repair is intentionally destructive :(
This one is an odd exception, that usually happens only with the early versions of a product.
Germany is rolling out an incentive to get products repaired. For example, repairs that can be done by yourself or in a repair café will be subsidized by €200 and repairs that require a professional or sending in, will be subsidized by 50% up to €200.
If the US gets into a full-blown trade war with China, i.e. massive tariffs on everything, I expect consumer electronics, clothing etc to become much more interesting to repair instead of replace.
I appreciate the intent, but.. Do you suppose time is free? and renewable?
Time that could be spent doing other things instead of learning how to repair, buying the tools, and repairing something that you could just hire someone else to repair, or buy anew?
As for environmental waste, figure out ways to make stuff out of more easily degradable stuff, or to reuse it, instead of _guilting_ people into repair.
A robust economy for parts and repairs would be ideal. Parts are (reasonably) obtainable, however, professional or even decent repair is about as reliable as a used car salesman.
This repair makes the case I've found more obvious though: we need microcontroller firmwares to be made available (I would argue as a public government archive honestly).
I've had several Yamaha amps blow micro-processors or DSPs, and on the older ones there were no firmware upgrades so you just plain couldn't do anything afterwards.
I've got the same problem with the controller for my heat exchange ventilation - microcontroller is dead, and while the chip is easy to get and replace, there's 0 chance I can source a replacement firmware for it.
Unfortunately, due to lobbying, it's now less of a trade but skill that corporations exploit for profit. In Western countries working on your own account is getting more and more restricted under guise of tackling tax avoidance (many tax authorities engage in anti-small business propaganda), so that the profit people generate is captured by corporations they have to work at if they want to pursue their "trade".
edit:
I find it fascinating that working class supports this, even though it is against their interest. It has a lot to do with crabs in the bucket mentality and the cultivated perception that venturing out of ones lane is wrong. Most people think they need to serve the rich masters and that is their calling and reject the idea that concept of class has been created to keep them in mental captivity.
It was hard to understand their purpose in the grand scheme of things though. Sometimes they were buried deep inside multiple layers of function calls.
I must say this is often true even when I'm looking at the source code. I think the vast majority of software out there is heavily overabstracted, and MCUs have gotten big and cheap enough (two ARM cores in an HDMI capture device!?) that this is appearing in embedded stuff too. This device shouldn't have needed more than one MCU.
Growing up in India about 20 years ago, we often repaired or renewed almost everything because our buying power was low and things were expensive. We used a lot of hacks, known as *jugaads*, to make things work. Even clothes were reused, with tailors doing *rafu* (patchwork) to extend their life. This was especially common in middle-class homes like mine.
My dad, who worked in a garment export house, used to tell me stories about how people in the West preferred disposable items and often opted for newer stuff, whether it was cars, gadgets, or clothes. At the time, I didn't understand this mentality. But now, with increased buying power and lower costs (thanks to China), we too tend to just chuck things away and get replacements.
I deeply admire people who don't give up midway and think it's easier to buy new. This type of persistence and resourcefulness is truly commendable.
Cost to repair has grown while cost for new has reduced.
If you can repair things yourself, and you can find the parts you need to repair things, then sure. But if it's something I've got to pay for someone's experience and wisdom, that's pretty expensive these days, at least where I'm living, and it's just plain hard to find people who repair things too; lots of signs for TV repair outside empty shops.
Thankfully I'm semi-retired, and am on a salary for part time work, so my marginal time has no dollar cost, so I can take a day to try to repair a dishwasher, and then another day to install a new one when it rebreaks a couple days later. A professional installer probably would have had the new one installed in an hour instead of my all day, but I didn't have to wait for scheduling, at least.
This was how things went for a long time in my region (western europe) as well, my parents grew up patching clothes and repairing stuff a lot. It's only in the past 50 years or so that consumerism has gone up and the quality and cost of e.g. clothing has gone down.
I've been doing maintenance on my motorcycle myself recently, it does take some small investments in some tools to get started (like a tool to undo the oil filter, although in hindsight a strap and a stick would do the job) and you need to source some parts and replacements (fluids, copper washers, but also replacement screws for the weathered brake fluid reservoir ones), but it's in the region of €100-€150 instead of the €1000 the garage quoted me for.
> It's only in the past 50 years or so that consumerism has gone up and the quality and cost of e.g. clothing has gone down.
The “quality” part is a big factor, cost optimisations and fast turnaround means it’s often not worth repairing things at all e.g. a fast fashion T designed to survive for a season (if it survives even a wash).
An other major issue is scams around price signals and brand degradation. It used to be you got what you paid for and some brands were known for quality, so you could pay a fair amount of money to a reputable brand and you’d get stuff worth maintaining and repairing.
But big groups and P-E have taken to “value extract” from brands, so they take a reputable brand and start white-labelling / cost-optimising, initially keeping prices in order to get maximum money for the moo their start selling instead of milk. Then they drop the price as understanding slowly spreads, until a once reputable brand becomes bargain-bin fare even to the general public.
There’s a similar issue around more bespoke products, which optimise for quality signals (e.g. external design and materials) and sell generic inner parts (or outright garbage) for top-shelf prices.
Then there’s the shuffling of 6 months brands on generic white label goods (amazon is absolutely infested with that, you’ll get the exact same product under half a dozen brands, and 6 months later most of those have disappeared).
It seems to me the real problem is that repairs are, in general, labour intensive. Few products today are sufficiently valuable that a repair is better than a new item.
Cars are worth fixing, a 10$ shirt is not, if you value your time. This only becomes more true as expertise becomes required to effect a repair, since you become less capable of repairing and the time of the repairer becomes more valuable.
Yes Reflecting on it, making things last longer had some great side effects. For instance, almost every woman in my family knew how to *rafu* clothes (1), and people understood how things worked under the hood of a car (like you my father did all the maintenance too). These skills were passed down through generations, becoming a part of our everyday knowledge.
I guess a lot of things aren't that simple or accessible as most of it is often a black box nowadays. But anyway, Skills like these not only saved money but also fostered a sense of self-reliance, resourcefulness and stuff your parents taught you as life skills.
(1) https://m.youtube.com/results?search_query=Rafu+clothes
I cannot help but winder if, as a part of 'Fix, don't toss' mentality, there is an attendant[1] additional tenacity present.
[1] Or a pre-requisite. Correlation, not causation and all.
The best tool I have bought in the last 3 years was a 3d-printer... It lets me make other tools - even if they aren't as durable as steel, I can design them chunkier, or print a new one if they break.
You can avoid the copper washers! The main reason to change them is under compression the copper work hardens to seal up. To get a good seal though you just need to re-anneal the copper so it's soft - heat it to cherry red and let it cool down. Takes about 5-10 seconds with a blowtorch - I've been doing it for oil changes on my car for several years now with no problems.
This is how it was for me growing up blue collar in the northeastern USA in the 80s. My father fixed everything in the house and the vehicles. I inherited my older siblings clothes, and my younger siblings inherited mine. My mother would hem pant legs shorter when we were young, and then let them back out as we grew older. If you wore a knee or an elbow out of clothes, it was getting patched.
This instilled some good and bad tendencies in me. I do almost all of the repairs around the house myself. I work too much though, so I don't always have enough time or energy. Even though I can easily afford it, I have a hard time paying someone else to do them. This means I live with broken stuff longer than I should.
I'd probably have more money if I spent that time working on side projects instead of doing maintenance and repairs.
> This is how it was for me growing up blue collar in the northeastern USA in the 80s. My father fixed everything in the house and the vehicles. I inherited my older siblings clothes, and my younger siblings inherited mine. My mother would hem pant legs shorter when we were young, and then let them back out as we grew older. If you wore a knee or an elbow out of clothes, it was getting patched.
Thing is, they were able to in the first place.
Forget about fixing a modern car. The electronics side is a mixture of "a datacenter on wheels", DRM and anti-tamper technology (sometimes enforced or heavily suggested by law such as in emissions control, sometimes by reality, e.g. "Kia Boys") and high-speed protocols instead of early age wires and relays that you could troubleshoot with a decent multimeter. The physical side is a ton of plastics designed to absorb crash energy and finely tuned metal alloy stuff (with the form also having crash safety implication) that your average DIY person cannot reasonably weld instead of plain old steel sheets. You can't buy a "reasonably repairable" new car any more because of the legal mandates and because you don't want it to be stolen by some kid having watched a YouTube or Tiktok video showing how to bypass the locks.
And clothing... patching a 1980s piece was possible, the fabrics had weight and structural integrity of their own. Nowadays it's extremely thin fabric everywhere that shreds itself after a few washing machine cycles. Try to patch it and you'll more likely than not find out that your very act of pushing a needle through it to apply the patch just causes the next rip to appear. You are still able to purchase better quality clothing technically but you end up paying like 4x the amount and it's still made in some Bangladeshi or Chinese sweatshop under horrible safety and employee rights standards.
If one has the proclivity, then: One can get into rather far into troubleshooting and (and ultimately repairing) common modern automotive electronics with an Autel rig that, adjusted for inflation, costs less than an Atari 2600 did.
I feel like it’s giving good money away when you hire someone to do work for you that you know you can do. You look at the markups for things and it gives you pause. Things like a valve or whatever. You can go to the Home Depot and get it for cheaper even when you include the cost of whatever tools you need to get.
But at some point you have to say, let’s just get someone to to it (the deck, the fence, the gutters, etc.) still it’s like zen and the art of motorcycle maintenance. There is some personal satisfaction in being self reliant.
In our past, this seems a community thing: Someone in your neighbourhood had advanced knowledge in welding, someone else advanced electricity, masonwork, clothes repair, ... They could coach other people to a basic enough level and help if unexpected troubles popped up. The community as a whole had knowledge and basic apprenticeships built in.
Today, repair is something you do on your own. Things like youtube are a great help, but the community aspect is lost.
When I bought my phone, it was on sale and so I got it new for $94. I dropped it and broke the screen. The replacement screen cost $106, more than the entire phone (just for the part, I do the work myself). I still did it anyway, for environmental/e-waste reasons and because repairing electronics is a fun hobby but the sad reality is, it often makes more financial sense to replace than to repair. That's not even going into the time lost and the money sunk on tools.
I grew up lower-middle class in America and to and extent I understand this.
Disposability is especially offensive when it comes to ~computers.
To have perfectly functional (from a hardware perspective) 10> year old smartphones become "e-waste" is absurd to me.
Even a cheap smartphone is a remarkable achievement in manufacturing and engineering. Its wild to think of something like that as "junk" even though it effectively is.
The quality of workmanship and repair has gone down drastically in India now. People have figured out how to make money by keeping things locked. Repair is just swapping out parts and cleaning or go for the next model. The mentality of people has also changed drastically with them changing products in short years. Consumerism has gone through the roof and there is an active disdain of people who preserve and get the product to function for a long time.
>Even clothes were reused
My mom repurposed old pants as bags (I used it to carry my books at college - did get a lot of odd looks, but that's what I could afford at that time). Even now, I cut pieces of old clothing to be used for cleaning purposes. Those habits die hard.
This was my attitude towards cars when younger. Always bought cheap used ones and fixed them. It frustrated me to no end seeing people fail at basic maintenance and cut the lives of their vehicles by half or more compared to a maintained one. (Stuff like not changing fluids or driving with the fluids leaking out, driving with slipping belts or broken suspensions, etc.)
This is fascinating. How were these usually jugaads obtained - did you more commonly get them from friends, family, shop owners, the Internet, or just (re)discover most of them yourself?
I agree that repairing is vastly preferable - I grew up in a poor communist country, where repairing and self-resourcesfullness was the norm.
However it's not the consumerist mentality driven by increased buying power - it's sheer economic sense - the time and effort is way better spent, economically speaking - by simply replacing. It's the lower price of goods that drive this thing.
My fridge broke the other week. I called the repairman who quoted a repair bill that was 10% MORE EXPENSIVE then buying a new fridge. Simply the time of the repair shop and the transportation from home to shop and back was more expensive them just buying a new fridge, chucking out the older one, and calling it a day.
My grandparents, nah, my parents would be horrified by throwing up a "nearly" good fridge. To me, it makes economic sense.
This is often due to the total costs being externalised (pushed off to others) and therefore not reflecting the true cost of the replacement nor the costs of (safe) disposal of the old unit.
Externalised costs such as emissions from manufacturing of new raw materials (metals, plastics, gases, etc.), transportation, disposal, and more.
Obviously it depends on what exactly fails. I've kept 'white goods' going for over 20 years despite:
I think sometimes repair-or-replace depends on one's state of mind. Figuring out what is wrong and how to fix can be frustrating but, equally, it can be extremely satisfying to realise you can do it and are no longer reliant on some mystical "expert" !Society as a whole in many countries is losing (or has already lost) the ability to be self-reliant and that lack makes people and communities generally more fragile.
Self-reliance is one of the drivers of hackers and tinkerers.
My economic theory is that the price reflects ecological costs fairly well.
At first glance it might appear that fixing would have a lower environmental cost. But the money spent will be spent by the repairman on things like international travel or whatever - and each of the things the money is spent on have environmental costs and externalities.
I don’t quite think so. I think we’re in a status quo that prevents/obfuscates a more efficient economic activity because it inflates GDP which makes the politicians and economists happy.
The magic number won’t go up much if you call somebody and they tell you what you need to do change the PCB and ship you the part(for a small markup price).
Don’t forget the emission offset credits somebody will pay for to dispose of that refrigerant liquid in the fridge!
I often develop feelings for the products I use (I know...). When I look at dishwasher I reminisce how many moments I had whilst standing next to it tirelessly working through my dirty dishes. I'll give it a tap. Sometimes I talk to it when loading like "Hey there, I got you some new stuff. Don't worry I'll feed you salt at the end of the week. Now I'll do your favourite program". Then once it finishes I say like "Oh what a great work you did there!" and so on. Then when it broke (the motor seized) I just wouldn't have heart to simply dispose of it. I sourced the motor and called in repair guy who installed it. It did cost me in total as much as I would pay for a new dishwasher, but I would never get the sense of feeling that I saved a friend.
I am so glad that I'm not the only one. Every time I shut the hood of a car after working on it, I give it a little loving pat-pat.
I have this inexplicable feeling, contrary to my usually rational self, that machines have a sort of soul and "feel better" when they're taken care of, and I feel like I'm letting it down when I extend an oil change/put off maintenance/don't take care of a problem I'm aware of yet. I don't really believe they have souls or anything; it's just a feeling I get.
Come to think of it, I do the exact same things with my plants too.
I can't explain it. I don't name my cars though.
I live in Sweden, coming from the Netherlands, both countries somewhere in the global top yet I still repair hardware, mend my clothes, repair the tractors and car, restored the 17th century farm we live on and extended it, built a barn and more. I intentionally do not make myself lust after the 'latest and greatest' of anything since I realise that such a lifestyle puts you on a treadmill, always running for the next treat. Hence I'm typing this on a computer from 2009 I got for free because the video card was 'broken' (27" iMac, a short stay in the oven later fixed the video card) connected to a second monitor I got for free because of a broken power supply (two capacitors later it worked again) which sits on a stand-up desk I got for free because of some trivial electrical defect (quickly fixed). In a way I still partake of the 'fruits' of that latest-and-greatest lifestyle, only with a decade or so of delay and without the compulsion to 'upgrade'.
Why do I do this? For a few reasons, most of them quite basic. I like fixing things. I get far more satisfaction out of using abandoned hardware which I have fixed myself than I get out of using whatever new gizmo I happen to lay my hands on because I know I can keep the former working (or find an alternative which I can get to work) while I do not know that for the latter. I like being self-reliant. With a soldering iron, a BGA rework station, a few old oscilloscopes and meters and a few decades worth of experience and scavenged parts I can keep things working for the most, design and build circuits to extend whatever is needed, etc. The advent of cheap and relatively open microcontrollers - the ESP series, Arduino, Raspberry Pi pico etc - has given a boost to the DIY electronics sphere which adds to the appeal of keeping older stuff working, e.g. I'm currently looking in to replacing the worn out control circuit and assorted switches of our 35 year old oven with something totally different and more functional, not because I can't get a new oven but because the current one works quite well apart from those switches. The same goes for the tractors and car, motorbikes (Russian Ural and Ukrainian Dneprs with sidecars), etc. There are no electronics in my tractors, they are purely mechanical. They can be repaired by anyone who knows how without the need for proprietary tools, dealer-only computer terminals and such.
Of course I could save a lot of time if I abandoned this 'life style' and just went with the flow, buying new clothes as soon as the old ones needed mending, buying a new computer every 3 years, a new car every 5 years, a new tractor every 10 years, a new dishwasher every 7-12 years, a new washing machine every 10 years, etcetera. I could stop cycling to the village and just take the motorbike, that would be much quicker after all. Think what I could do with all that time saved:
- instead of cycling to the village I could spend time at a sports school for exercise
- instead of fixing that computer (and learning a bit more every time I fix one) I could watch some series on some streaming service
- instead of mending that hole in my trousers (using the Elna sewing machine I got for free because it was 'jammed', took me all of 5 minutes to unjam it) I could browse the web looking for some new trousers - something I'd have to do every few weeks since my clothes somehow seem to acquire holes quite easily, why would that be?
- instead of building that barn I could be working a few more months to pay someone else to build me a barn
- instead of gaining self-reliance I could make myself become more and more dependent on outside sources and 'experts'
Well, thanks but no thanks, I'll just keep on mending my own stuff simply because I can and I like it that way. Here, in Sweden, in the land of plenty.
If you're interested in extending the service life of that old iMac, I recommend OpenCore Legacy Patcher. It lets you run newer versions of macOS on old hardware.
https://dortania.github.io/OpenCore-Legacy-Patcher/MODELS.ht...
The "consumerism" mindset is a luxury created by a strong economy and high upward mobility. An unfortunate side effect is that as repair & reuse became less desireable, so too did repairability and logevity as features. Now that economic growth in the US has stagnated (particularly for the 99%), it's becoming aparent what we lost as nothing is made to last and companies are unwilling or unable to offer repairable, long lasting products. So even those who go out of their way to repair their property encounter a myriad of roadblocks.
>A light bulb went off in my head. Yes, these chunks of data in the flash chip were pictures! But not in the conventional sense. They were describing the various LED animations. Each set of 16 bytes is an animation frame written to the 14 LEDs. In the example above, a single red LED turns on and six of the seven white LEDs do stuff. The last two bytes in each row are a delay time before moving onto the next frame. The first 16 bytes looked kind of weird though. They looked like they possibly contained some header data for the animation or something.
I have independently invented this format, albeit in text instead of binary, for animating my Hue lamps. Animations in my version look like this:
where the first value is the number of milliseconds to delay before moving to the next line, then each led channel's brightness (out of 100) is represented on each line. A line that starts with anything but a number is a comment, and any number of spaces (1+) are allowed between channels.I'm tickled to see a similar (binary) format in a commercial product. It really is quite obvious, but still interesting to see.
This is like tracker software, or player piano rolls for LED animation.
https://en.wikipedia.org/wiki/Piano_roll
What a humbling writeup. This is real fullstack engineering.
Humbling all right. I think it would take me weeks to achieve this - and that's assuming I had this article to start from!
I suspect it may have taken the author weeks as well, given they indicated waiting for chips from China and that there was communications with Elgato's customer support. This writeup looks like it's the report of a few weeks if not months of on and off work.
I'm the author of this article and you're absolutely correct! This was a long, drawn-out project. For some context, I ordered the replacement regulators in February. The new LED driver chips were ordered in March, so that was around the time that I actually had the failed hardware fixed. Then everything sat idle for months. The firmware reverse engineering to figure out the LEDs was several weeks of on and off work in my spare time.
This type of thing is definitely not something you can just figure out in a couple hours (or even days).
Your writeup is inspiring.
I most appreciate the honesty about mistakes and failures and dead-ends. Too often wasted effort and small failures are elided from articles and instead you get something that is written retrospectively as though everything went perfectly...
Thanks! I wouldn't want to write it any other way. I think it helps convey just how much work it really was.
How do people maintain motivation on projects like this? Repeatedly having to order parts and wating for them to arrive, especially the one from China, make me abandoned an unfortunate amount of projects.
The time, and also the cost of ordering these parts. I know each is like a few cents, but you don't know all the parts you will need beforehand. So if I counted correctly he ordered at least 3 times.
It's fascinating nonetheless.
Would love to see Elgatos response to your writeup :-)
Same here. I can totally imagine the excitement in their engineers eyes after reading this fascinating writeup, paired with their managers anger while they ask for a better way to protect the company's IP.
It’ll have to come from Corsair who now owns Elgato. IMHO, they’re a ‘conventional’ company with stock listed on the exchange. If they respond I will be pleasantly surprised.
Buy one share of their stock, then call investor relations and ask them why a hobbyist was able to fix the LED bug that their support claimed didn't exist while their whole expensive engineering team was not.
>while their whole expensive engineering team was not
Question is maybe not if they could, but if they were allowed to. If the project is already out of development and a bug is detected, sometimes there's only a barebones team left that barely gets the time to do basic maintenance.
Now i am curious if they know what causes their LED troubles or what causes the underlying memory corruption.
Elgato stuff is cheap but horribly unreliable in my experience. Thanks for the writeup, it's showing why that is very detailed...
The flashing and corruption problems point to a very poor software engineering culture.
That is awfully close to blaming the engineers.
You don't know what kind of pressures they are under to deliver.
I know of significant compromises in all the software I have written, and I would make hardware mistakes if I had taken the electronic engineer path. Sometimes we have to prioritize and I've never had an unlimited time budget on any project (not even my own).
Projects with no issues are mostly dead!
A poor engineering culture usually exists because management broke it, rather than because the engineers failed to create it. Of course I have no insight into how Elgato is run, so I am not pointing fingers. I merely state that there is certainly evidence that their engineers are not focused on quality.
Well high pressure to deliver is sort of a poor culture isn't it? I agree we should be forgiving of others mistakes, but I also think we ought to strive to develop better solutions that lead to fewer bugs at any given velocity.
How does the corruption point to a software engineering issue? What could cause corruption and how could it be avoided by better software engineering?
It's a culture issue when such cases don't get caught in testing and support staff either doesn't know about it (=the scripts are bad), doesn't get told that there is a workaround (because clearly there is), or (the worst of the possible options) gets told to act like everything is fine.
It's a race condition which involves physical interaction with the product, which only occurs during a very rare operation, which looks pretty much identical to a genuine hardware defect.
This isn't something you can just trivially unit test. If you don't see this happen multiple times during initial hardware development, you are never catching it. A single failure of a prototype can easily be attributed to a manufacturing defect - especially if it was hand-soldered.
Once it's in the field replacing the 0.01% of units suffering from random issues under warranty is far cheaper than having an engineer spend weeks trying to diagnose every single weird failure mode. Unless it affects a number of units, it's just not worth the money. You have to consider that support doesn't get a "the LEDs stopped working when I unplugged the device immediately after flashing it" message, they just get "the LEDs don't work". Support scripts are made for horses, not zebras.
> Once it's in the field replacing the 0.01% of units suffering from random issues under warranty is far cheaper than having an engineer spend weeks trying to diagnose every single weird failure mode. Unless it affects a number of units, it's just not worth the money.
It does affect a number of units, that's the point, and it's serious enough that the brand image is suffering. Just google "elgato unreliable" - tons of results, and when the top result is a Reddit post in the official Elgato subreddit of all places literally titled "My elgato experience has left me with nothing but hate in my heart", all alarm sirens should go fucking off.
To top it off, Elgato used to be a German brand right out of Munich [1]. German products used to be noteworthy for top-notch engineering and reliability...
[1] https://www.reddit.com/r/elgato/comments/18k92zy/my_elgato_e...
[2] https://en.wikipedia.org/wiki/Elgato
It’s sad because their stuff, like the key lights and the stream deck are neat and really useful. and the software has all the features you’d want and you can make your own plugins for it and they document the process for making the plugins. All really cool.
But in practice I’m constantly having to power cycle my key lights and having to restart the stream deck app on windows.
Btw I switched from using their app to bitfocus companion instead. Still lets me control my stream deck but no more elgato Corsair bloat
I maintain a lot of their stuff for my coworkers. It's nice looking, and easy to understand, but they also have a lot of things that are kludgy clunky or strange that don't make any sense.
Their ring light, for instance, can't be manually turned off using the shoulder buttons. It can only be set to a low dim. Why not? Makes no sense.
Their FaceCam needs a high bandwidth usb 3.0 port all to itself. Won't reliably work over a hub. Why not? I mean, give it a low bandwith 720p mode or something.
The Mic has a headphone jack, sure, no problem, but why does it install 8 audio devices during the install? Only install what is needed!
Their USB capture cards have issues staying connected unless they are plugged directly into the laptop/desktop motherboard, and even then they flip out every few hours.
These issues have ensured that I won't buy any of their stuff. I've recreated an entire elgato stack that would have cost $500+ for like $90 in cheap chinesium crap and it works perfectly.
If you don't have a thermal camera, you can also paint rubbing alcohol onto the board and watch where it evaporates the quickest. Obviously, this only works for low voltage stuff...
Thanks for the writeup! It was great as you explained the tools and rabbit holes you went it to.
The reason it likely goes into bootloader when the lock bit isn't set is that it's likely using this as a "flashed successful" signifier.
Thats explicitely stated in the post:
Interestingly I discovered that if the chip is set for unprotected mode, the bootloader actually waits around for update commands instead of booting the application firmware. So Elgato is using the locked/unlocked bit as a way of signifying whether it should stay in the bootloader or not, which I find to be a little bit weird. That completely explained why my second HD60 S stopped working after I installed my unlocked firmware — it was stuck in the bootloader waiting forever because the chip was unprotected.
Really enjoyed this write up. You both had the skill and the patience to even buy a working version to fix the broken ones!
Great work! I find it bizarre that they even bothered with this wild SPI Flash muxing thing if all it does is drive some LEDs. Seems massively overengineered.
amazing work! ultimately it comes down to a firmware bug - I would not have had the fortitude to continue when it came to debugging
Great writeup, with very handy links!
What a well written reverse engineering project :)
The ability to explore & chase down truths is what makes us divine, is the aspect of us that builds & enhances. This guy walked a deep deep spiritual path to exploring something of the mundane world.
It's just so sad, so frustrating, so hellish & infernal that devices do resist us. Computers & devices should be amplifying & radiating the truths they are made of. But humanity is trapped in shells of their own making, in descending consumerism that rejects engagement & learn ability. Computers, alas, especially so, are often the prime obfuscater of truth and understanding! What a vast disuse of so much potential!
I hope we see some break through currents, of machines that are open, emerge in my lifetime. Well done, this dude Doug.
If the right to repair movement succeeds fully, OP would have had access to the complete source code and hardware schematics of this device, and the fix for his flashing lights probably would have been a 5 line shell script that someone else had already put on github.
> would have had access to the complete source code and hardware schematics
Forcing every company to open source their products is a pipe dream. This is never going to happen.
Even if one country passed such a law, manufacturers would move their production to a different country. The products would then be imported and redistributed as foreign products, circumventing the law.
People would have said the same 100 years ago about forcing food companies to reveal their ingredients list. Yet now it is required in almost every country, and doesn't appear to have been that detrimental to the food industry.
Ingredients listing don't tell everything about how it is made, etc.
A closer analogy might be detailed API specs.
Hardware schematics and full source code doesn't tell you everything about how a piece of hardware is made either.
How was the hardware tested? How was it calibrated? How was it assembled?
An ingredients list tells 90% of the story about how food is made, and in most cases an expert could guess the remaining 10% to get a decent result. Likewise, hardware design+code tells 90% and an expert could figure out the remaining 10%.
The government could totally require all consumer products have published source code. (published source code != a license for others to use the code)
For electronics it would probably show the part number and position of each component on the PCB, without showing how they are connected.
I agree with the sentiment, but:
To what level would the "source code" have to be published? Chip-level? Should I expect HDL code that allows me to reproduce the microcontroller? If not, expect a bunch of gadget companies to pay cypress or whoever to make "custom" chips with their firmware burned in. After all, what's the difference at that point between HDL and firmware?
If yes, expect most companies to simply refuse, and not authorize their chips for sale in the US. International IP / licensing agreement would make it literally impossible for them to comply without being sued into oblivion by the Taiwanese company they licensed the IP from.
I think stuff like design files and source code should be held in escrow by the government. If you continue to provide replacement parts to customers, the information remains protected. Once you stop providing reasonably priced replacements ("reasonably priced" = less than the cost of the product), the information gets published so others can reproduce it.
We need to encourage our kids to repair stuff. It aids the environment (less e-waste) and teaches them a valuable trade. I know that's it's typically cheaper to replace than repair, but to me that's a market failure.
> market failure
Industrialization has massively optimized the "hot path" for any kind of production, largely by taking out the humans. It is incredibly cheap to manufacture consumer electronics - because everything is standardized and predictable and amenable to machine operation. This is what market success looks like.
The downside is that as soon as there is a deviation from the process, it gets more expensive.
It's worth thinking about what happens to repair inside the factory. Some percentage of units will be coming off the production line defective and fail initial QA. This is usually a very low percentage, well below 1%, because the entire economics of the factory depends on not having to do any rework. Even there, the technicians will take a look, determine if it's something that can be fixed quickly, or is a novel kind of failure, and if not just throw it away right there. Why? Because while you're standing there looking at it, another hundred have come off the production line. The broken one in your hand is not a unique snowflake.
At the start of a run though, this is an interesting and important job, because any failure is a novel failure. The first batch through the process should have high yield, but it might not, and then you stop the line ( https://mag.toyota.co.uk/andon-toyota-production-system/ ) and figure out what's happened (something misaligned? Defective inputs? Material problems? Design issue?)
The first batch will often get reworked and sent out as demo units.
What happens when a unit is defective and not easily repairable, maybe because the PCB itself was already defective?
Do technicians salvage the high value components, like the CPLD in this case, off the board? That chip alone is probably worth 15$.
As other commenter says, spending human time to save a $15 chip is uneconomic. This only happens if they're especially expensive parts or (as during COVID) supply is short and you can't just buy some more.
(COVID shortages saw the opposite phenomenon in a few places: working consumer electronics being bought and stripped for a particular part critical to something else, then the rest thrown away!)
Remember that after you've hotgunned it off the board and cleaned the solder you have to re-ball BGA parts. Again, a process that's cheap in the original manufacturing line and very hard to do by hand. It also means the part has been through more thermal stress which will shorten its life. You don't want to have to rework a unit again if you put a recycled chip in it which fails.
The engineering time to "salvage" and test the 15$ chip, and the cost of downtime reintegrating the part to the production line is probably worth more that 15$, so unless the lack of that specific part is a bottleneck, probably no?
Yeah, probably, but I do wonder where it starts to make sense to desolder a chip and give it another try.
Surely at the point of high end GPU Boards one would invest considerable amounts of time and money to salvage a chip, which could sell for thousands of dollars.
I think you're right that for high end chips, it is worth the time to salvage it[1]. But I think that's the vast minority, like you said, most stuff being manufactured isn't at the cutting edge of consumer tech like GPUs are.
[1] They show a bit of the RMA/failed unit process towards the end of this surprisingly good GPU factory tour from Linus Tech Tips. There's some discussion of de-soldering and testing retrieved components. https://www.youtube.com/watch?v=GS35VHEfFDU
Not the user you replied to from my experience usually it's not worth it due to several reasons:
- desoldering the chips takes time and is a manual process ($), with risk of tearing off a pad or bending leads. In case of BGA ICs reballing is needed to reuse them. - components are usually not rated for a lot of reflow (heat/cool) cycles, and some are moisture-sensitive and may crack if they have managed to absorb moisture - you usually end up with some solder and flux left on the IC, which can cause issues - ICs come on tape for feeding into automated pick-and-place machines, so you would need to feed and mount them manually ($)
And if you only realize you have damaged the IC after mounting it on the new board you end up having to rework it again ($).
Sure, it might be worth it if the chip is really expensive or hard to get, or you're soldering everything by hand anyway, but usually the math just doesn't work out.
Most devices nowadays are unfortunately not only not repair-friendly, they are straight impossible to take disassemble. They are welded, soldered, glued together, and any attempt at a repair is intentionally destructive :(
This one is an odd exception, that usually happens only with the early versions of a product.
Germany is rolling out an incentive to get products repaired. For example, repairs that can be done by yourself or in a repair café will be subsidized by €200 and repairs that require a professional or sending in, will be subsidized by 50% up to €200.
If the US gets into a full-blown trade war with China, i.e. massive tariffs on everything, I expect consumer electronics, clothing etc to become much more interesting to repair instead of replace.
I agree with the sentiment, but in this example, how much domain expertise was required?
I appreciate the intent, but.. Do you suppose time is free? and renewable?
Time that could be spent doing other things instead of learning how to repair, buying the tools, and repairing something that you could just hire someone else to repair, or buy anew?
As for environmental waste, figure out ways to make stuff out of more easily degradable stuff, or to reuse it, instead of _guilting_ people into repair.
To repurpose a sentence from somewhere else, repair is only free if your time is worthless.
(Imagine trying to contract someone to do the OP's level of thorough work; it would be a five-figure dollar sum)
A robust economy for parts and repairs would be ideal. Parts are (reasonably) obtainable, however, professional or even decent repair is about as reliable as a used car salesman.
This repair makes the case I've found more obvious though: we need microcontroller firmwares to be made available (I would argue as a public government archive honestly).
I've had several Yamaha amps blow micro-processors or DSPs, and on the older ones there were no firmware upgrades so you just plain couldn't do anything afterwards.
I've got the same problem with the controller for my heat exchange ventilation - microcontroller is dead, and while the chip is easy to get and replace, there's 0 chance I can source a replacement firmware for it.
Unfortunately, due to lobbying, it's now less of a trade but skill that corporations exploit for profit. In Western countries working on your own account is getting more and more restricted under guise of tackling tax avoidance (many tax authorities engage in anti-small business propaganda), so that the profit people generate is captured by corporations they have to work at if they want to pursue their "trade".
edit:
I find it fascinating that working class supports this, even though it is against their interest. It has a lot to do with crabs in the bucket mentality and the cultivated perception that venturing out of ones lane is wrong. Most people think they need to serve the rich masters and that is their calling and reject the idea that concept of class has been created to keep them in mental captivity.