Comment by nasanu on 27/01/2025 at 01:08 UTC

-13 upvotes, 4 direct replies (showing 4)

View submission: Massive Failure on the Product

View parent comment

Did you read? The issue was with the prod database. Do you test on prod? If not then this could also happen to you.

Replies

Comment by AGRYZEN at 27/01/2025 at 01:10 UTC

16 upvotes, 1 direct replies

OP has added context of the issue since my comment - but also, what?

Comment by neb_flix at 27/01/2025 at 02:18 UTC

11 upvotes, 2 direct replies

How inexperienced are you that you think that testing against a production data source must only happen once you deploy a client to a user-facing production environment?

First off, the fact that no one realized that 95%+ of their users would not be able to register at launch due to them already having entries in a table for these users is a crazy misstep, both from a software design perspective and a QA perspective. Knowing that they had to have had recently migrated that data to the production DB, why did no one on the team call out that they would not be able to register if those users existed in the given table? Are there no processes that aid for this communication across the team (a la Pull Request?)

Secondly, i'm having a hard time thinking why this wasn't an almost immediate remediation if what the OP said about the issue is accurate. Any experienced dev involved in the project should have the ability to quickly drop the table, or remove the offending records (i.e. before a certain creation datetime). If you are launching a product and you know that you are losing users & leads every minute that the product would be down or not working properly, a competent team would make sure that they are enabled to fix these kind of trivial issues (i.e. brokered the appropriate access to prod databases/data sources).

Comment by notsooriginal at 27/01/2025 at 01:15 UTC

3 upvotes, 0 direct replies

You said the same type of database twice! /s

Comment by manys at 27/01/2025 at 02:53 UTC

2 upvotes, 1 direct replies

Never test on production! The entire point of 'staging' is to have the same schema as production, it's not "development (serious)."