10 upvotes, 2 direct replies (showing 2)
View submission: Massive Failure on the Product
How inexperienced are you that you think that testing against a production data source must only happen once you deploy a client to a user-facing production environment?
First off, the fact that no one realized that 95%+ of their users would not be able to register at launch due to them already having entries in a table for these users is a crazy misstep, both from a software design perspective and a QA perspective. Knowing that they had to have had recently migrated that data to the production DB, why did no one on the team call out that they would not be able to register if those users existed in the given table? Are there no processes that aid for this communication across the team (a la Pull Request?)
Secondly, i'm having a hard time thinking why this wasn't an almost immediate remediation if what the OP said about the issue is accurate. Any experienced dev involved in the project should have the ability to quickly drop the table, or remove the offending records (i.e. before a certain creation datetime). If you are launching a product and you know that you are losing users & leads every minute that the product would be down or not working properly, a competent team would make sure that they are enabled to fix these kind of trivial issues (i.e. brokered the appropriate access to prod databases/data sources).
Comment by TheScapeQuest at 27/01/2025 at 08:54 UTC
2 upvotes, 0 direct replies
In high pressurised environments, stupid mistakes can happen.
I used to work (contracted) to a major UK telecoms provider. We had 3 major releases over a weekend (6am release on Friday, Sunday, Monday). There was a last minute legal challenge against some of the terminology we were using for the Monday campaign so we had to very quickly fix it. We only tested the "organic" journey, rather than through affiliate sites. Come about 10am on Monday we realised sales were massively down because we broke affiliate journeys (about 90% of sales).
Overworked employees cannot be trusted.
Comment by nasanu at 27/01/2025 at 06:23 UTC
-5 upvotes, 1 direct replies
Wtf are you on about? Nobody just pushes code to prod to test.