💾 Archived View for wilw.capsule.town › log › 2021-12-11-nextcloud-object-storage.gmi captured on 2023-04-19 at 22:42:05. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Some people may remember my quest a few months back to find a good alternative to Google Photos [1] for image storage and backup.
At the time, I talked about Piwigo, Mega and pCloud as potential candidates. I also briefly touched upon Nextcloud in that post - a service I use (and self-host) anyway for all of my other storage needs, but I did not consider it further due to the high cost of the associated block storage required to house a large number of images.
In this post I want to explore a different approach to storing files using Nextcloud: one that uses _object storage_ (instead of block storage) as a backend.
This is a fairly obvious point these days in the cloud world, but I thought I'd include it in case the reader isn't familiar with these terms.
_Block storage_ is a facility offered by cloud and infrastructure-as-a-service (IaaS) providers. It is a system that enables you to create _volumes_ that can then be attached and mounted by your cloud virtual machines. Once attached, they can be formatted with a filesystem of your choice and used just like any other filesystem mounted by your machine.
Block storage volumes can often be detached and re-attached to other instances, or even persist without being attached to any instance. Most modern providers use high-performance SSDs to power this facility, making it a great option for data-processing on cloud servers.
_Object storage_, on the other hand, is more of a _service_ offered by the cloud provider. Users can create containers known as _buckets_, and in these store any number of arbitrary files ("objects"). Buckets don't get attached to instances, and the objects they contain can be accessed completely independently over the web or via SDKs.
As such, object storage can be super useful for storing and retrieving user-uploaded content in a service, or for storing large collections of arbitrary data.
From the viewpoint of this post, the key difference between the two is the _pricing_. I use Linode as my cloud provider of choice, and - to give you an idea of the scale - 1TB of block storage [2] would cost you over $100 a month. The same amount on object storage [3] would be $20 a month.
Another factor to consider is that block storage must be pre-provisioned. I.e., once you create the volume you begin to be charged even if the volume is unformatted and empty. In object storage the cost scales depending on the amount of data actually contained by your buckets.
Based on the above, object storage is naturally a sensible choice if one were to choose out of the two for storing large volumes of images, since it's the one that competes the most with other commercial offerings in terms of pricing.
Neither Apple iCloud [4] or Google Photos [5] offer a 1TB option, but their 2TB storage options are priced at around $10 a month.
Whilst this is essentially quarter the price of the Linode object storage pricing (which, in itself is great), the commercial offerings must also be pre-provisioned (i.e. you pay that amount even if you only upload one photo). You also become more locked into their relevant walled gardens, with all of the limitations I spoke about previously [6].
As such, I believe that a system based on your own managed block storage solution is a viable alternative.
Nextcloud [7] offers a great web interface and mobile app for accessing your files and viewing stored images. The mobile app also has an "auto-upload" facility for photos, which is super useful.
This, combined with the object storage, provides a more cost-effective way of managing files and photos stored in your Nextcloud.
Some people have reported issues with Nextcloud when viewing large folders of images (since the thumbnail-generating task is quite CPU-intensive). However, I am keen to give this a go to see if it could be a viable option.
To set-up a Nextcloud to use object storage I recommend creating a fresh new instance, since changing your existing one will make your previously-stored files unavailable.
In this post I will use Linode's object storage as an example, but the process should be similar for any S3-compatible backend (e.g. Amazon S3, DigitalOcean Spaces, Backblaze B2, etc.).
To begin, login to the Linode dashboard and navigate to the "Object Storage" page. Then click "Create Bucket", give your new bucket a name and choose a geographic region for its storage.
Next, navigate to the "Access Keys" tab on the "Object Storage" page and click "Create Access Key". Give your key a label to remember it by, and select to give it limited access such that it can only read/write to the bucket you just created.
When created, a one-time dialog will popup to display the details of the new key. Make a note of both the "Access Key" and the "Secret Key", and then close the dialog.
At this stage, you should now have:
You can now close the Linode dashboard.
I use Docker to run my services, so I recommend creating a new `docker-compose.yml` file with these contents:
version: '2'
services:
nextcloud:
image: nextcloud:23
volumes:
- ./nextcloud-data/storage:/var/www/html
restart: always
Feel free to add any network configurations as required (e.g. to an Nginx host for TLS termination) and then bring up your instance: `docker-compose up -d`. You can also specify a database container in your compose file if the default SQLite option isn't appropriate for your needs.
Once up and running, Nextcloud should create its files and directories in the `nextcloud-data` directory. To configure your instance to use object storage for its primary storage, you'll need to add some lines to `nextcloud-data/storage/config/config.php`:
<?php
$CONFIG = array (
... // Omitted for brevity
'objectstore' => [
'class' => '\\OC\\Files\\ObjectStore\\S3',
'arguments' => [
'bucket' => 'BUCKET_NAME',
'key' => 'ACCESS_KEY',
'secret' => 'ACCESS_KEY_SECRET',
'hostname' => 'eu-central-1.linodeobjects.com',
'port' => 443,
'use_ssl' => true,
],
],
...
);
You'll need to change the `BUCKET_NAME`, `ACCESS_KEY`, and `ACCESS_KEY_SECRET` to the values you noted earlier. You may also need to change the `hostname` if you use a different storage region or a different provider.
You can now save and exit the file.
To test the set-up, navigate to the URL you've configured for your instance. If this is the first time you've visited it there will be a few bits of setting-up to do, which Nextcloud will guide you through.
Once you're through and logged-in, you can try adding files using the web interface and then use the Linode dashboard to verify that Nextcloud has created new files in the bucket. Note that the file names and extensions will be changed, since Nextcloud manages metadata and naming in its own database.
If something goes wrong then just check the logs of your running Nextcloud container: `docker-compose logs`.
Finally, when you're ready, you can try importing your photos or enabling the Nextcloud app auto-upload feature on your phone. This feature can be found under the app's settings.
I recommend trying it for a while and seeing how you get on with regards to performance and stability.
In this post I've covered the advantages of object storage, and how this mechanism can be applied to Nextcloud to provide a self-hosted alternative to Google and Apple photos.
Personally, I am still not 100% on the performance of Nextcloud in terms of photo-viewing. However, I would like to give it a chance and it could be viable anyway as a secondary backup to an existing primary system.
Google's own image search and navigation is so convenient that creating a real competitor is difficult. However, I am keen to hear how others get on in this space!