💾 Archived View for dioskouroi.xyz › thread › 29376820 captured on 2021-11-30 at 20:18:30. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
________________________________________________________________________________
Dear soheil,
I happen to need around few thousand in GCP credits in order to egress around 30TB of data from our cloud buckets. It's training data, training logs, model snapshots, and pretty much all of our ML history (
).
The real tally is actually 83TB, but around 30TB of that is incremental model snapshots which we don't need to download (turns out, you can generate a lot of data when you're using TPUs for training), and around ~30TB is whatever-data that no one cares about / duplicated data across regions. (To use TPUs in EU and US, you need to place the training data in both EU and US.)
Normally I wouldn't ask, but we've been playing "hot potato" with this data for several months now, since our last credits ran out. We've basically been making new GCP accounts (where you get $300 credit) and then letting it sit there for a month (because they give you a month before deleting it, even if the credits run out).
If you'd be willing to help, it would be quite simple: I would transfer the buckets to your account, and then download the data to a Hetzner server. I'd then make the data publicly available, such as it is. You can DM me on twitter (
) or email me (see profile). Thanks for considering!
EDIT: For an example of how we've amassed terabytes of training logs, you can see some of our BigGAN outputs here:
https://twitter.com/search?q=from%3Atheshawwn%20biggan&src=t...
Do these snapshots or models even matter anymore? Are snapshots from training some image generating model really useful to anyone and arent there already newer and better models out?
You are allowed to run BOINC[0] and folding@home[1]
[0]:
https://old.reddit.com/r/gridcoin/comments/7vhku5/fyi_we_are...
[1]:
https://github.com/GoogleCloudPlatform/terraform-folding-at-...
This is probably the best answer, aside from "don't contribute to the environmental waste".
I think it’s even better than not contributing to the environmental waste. I believe helping to find cure for many of the deadly diseases will save more lives than cause harm to environment by running some gpus/cpus. Like way more.
I suppose it's the distinction between finding the Haber-Bosch process good or not.
That is the only „valid“ answer IMHO. Spend all on BOINC and do something for the world.
10000% this
We need more computing resources dedicated to scientific work rather than the (in my biased opinion) bullshit consumer apps that don't actually help anyone with anything besides making their owners assloads of money
It really shouldn't, but stories like this irks me a lot because I have been trying to get credits for my bootstrapped startup for sometime and there is just no way to get them unless you are part of a accelerator or took funding from well known investor.
Of course, Gcp, Aws, DO don't owe me anything and these programs must be successful based on their data but still shows the irony of the world where resources aren't matched well against needs.
You can join startup school and get them.
Don’t use it if not needed. On the contrary, it contributes to the environmental waste.
You mention training models... Do you want to donate them to ML researchers or students?
Maybe people can use these on our platform (
) which is on GCP and GKE. The platform provides real-time collaborative notebooks so people can train, track, package, deploy, and monitor machine learning models on the users' Kubernetes clusters (AWS, GCP, Azure, Digital Ocean, etc. So if you have credits elsewhere, this may be doable).
You could transfer your credits to us, we'll generate a special link, then you donate that link to people or entities who, when signing up with the link, could use your resources.
How about seeing how big of an index of the web one can build for 25k?
I am inspired by this:
https://news.ycombinator.com/item?id=28550764
I was thinking about building a domain name index, this would give the project a nice deadline and budget :-)
If you want to collaborate, my email is in my profile.
How did you manage to get that much? I’ve been trying with no avail to get some decent credit to run my side project. I’m playing the whack-a-mole game to signing up new account every time my $300 runs out.
How about running heavy computations for somebody else for some $?
"ROI"
Maybe you can mine some Ethereum.
Not sure if that breaks their TOS, you'd have to check to see that this is allowed.
It certainly breaks their TOS.
Why?
Setup a cloud gaming rig and max it out :D
Can't you just buy some really long term upfront storage contract?
You'll always need storage.
No, credits can't be used for purchases beyond the credit expiry. This is how AWS[0] limits it:
> Promotional Credit will be applied only to offset eligible fees and charges incurred during or following the billing cycle in which you apply the applicable Promotional Credit code to your AWS account. Promotional Credit will be applied only to the specific Services designated by your AWS contracting entity (collectively, “Eligible Services”). Promotional Credit will not be applied to any fees or charges for Amazon Mechanical Turk, AWS Managed Services, Ineligible AWS Support, AWS Marketplace, AWS Professional Services, AWS Training, AWS Certification, Amazon Route 53 domain name registration or transfer, any Services for mining for cryptocurrency, any other Services as may be designated by your AWS contracting entity, or any upfront fee for any Services such as Savings Plans and Reserved Instances (collectively, “Ineligible Services”). For purposes of this Section 1, “Ineligible AWS Support” means AWS Support (as described on the AWS Site) that is at the Enterprise Support level.
Can't find the relevant GCP doc, but I'm sure that is similarly limited.
[0]: