Moving media files to self-hosted Minio S3 storage

In this guide, we will create a self-hosted MINIO S3 bucket called "mastodata" at the address "https://media.your-server-social:9000" and then proxy the bucket address as "mastodon-media.your-server.social" for use by Mastodon.

(When copying over instructions, watch out for the extra blank space in https: // prefixes.)

Minio server


Configure machine & install certificates

  1. sudo apt update && sudo apt upgrade -y
  2. Install & configure fail2ban
  3. Open firewall ports: 80 (http), 443 (https), 9000 (S3 API), 45135 (Admin console, you can pick a different port in your minio configuration)
  4. Ensure DNS entry for media.your-server.social is there
  5. sudo apt install certbot
  6. sudo certbot certonly --standalone -d media.your-server.social --staple-ocsp -m admin@your-server.social --agree-tos


See also:


Install Minio server with HTTPS

Create user & path
  1. adduser --disabled-login minio-user
  2. mkdir -p /storage/location/for/minio
  3. chown minio-user /storage/location/for/minio


Link certificates
  1. su - minio-user
  2. mkdir -p .minio/certs
  3. ln -s /etc/letsencrypt/live/media.your-server.social/fullchain.pem /home/minio-user/.minio/certs/public.crt
  4. ln -s /etc/letsencrypt/live/media.your-server.social/privkey.pem /home/minio-user/.minio/certs/private.key
  5. exit


Install Minio, but don't start the service yet

Allow MinIO to use port 443:
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/minio

There are 2 configuration files:

  • Systemd service: /usr/lib/systemd/system/minio.service
    • Follow instructions from the installation link above. Important: Use systemctl edit --full minio.service to edit, otherwise your changes will be overwritten when you update Minio.

  • Server config: /etc/default/minio
    • Set server URL with https: MINIO_SERVER_URL="https://media.your-server.social:9000"
    • Set fixed port for console to have it permanent in the firewall: MINIO_OPTS="--console-address :45135"


Test with:
  • sudo systemctl restart minio
  • systemctl status minio


The status output should contain 2 https links:

  • S3-API: https: //media.your-server.social:9000
  • Console: https: //media.your-server.social:45135


If it's working, enable the service:
sudo systemctl enable minio

See also Generate Let’s Encrypt certificate using Certbot for MinIO



Configure MinIO bucket

Visit the MinIO console at media.your-server.social:45135 and log in with the MINIO_ROOT_USER from the server config file at /etc/default/minio.

  1. Go to Settings
  2. In Region, enter a region
  3. In API, double-check that "Cors Allow Origin" is set to "*"
  4. Create a bucket "mastodata"
  5. Configure access using the instructions from joinmastodon.org
  6. Visit the "mastodon" user, then Service Accounts. Create an access key there - this will be used by the Mastodon server to authenticate.
  7. Visit the "mastodata" bucket and add Anonymous access for "readonly"
  8. Double-check the bucket's custom policy to ensure that it won't allow the listing of objects


Mastodon server


Copy the media files

Before you start copying, consider deactivating all relays so that you won't accumulate too many new media files while you work.

Clean up the Mastodon cached media:

  1. su - mastodon
  2. live/bin/tootctl media remove --days=2
  3. live/bin/tootctl preview_cards remove --days=2
  4. live/bin/tootctl media remove-orphans


While that's running, install the media copy tool.

Set an alias to access the S3 bucket with the copy tool:
./mc alias set media https: //media.yourserver-social:9000 accesskey secretkey

Once the file cleanup has finished, you can do your first round of copying:

./mc cp -r --md5 /home/mastodon/live/public/system/site_uploads media/mastodata
./mc cp -r --md5 /home/mastodon/live/public/system/custom_emojis media/mastodata
./mc cp -r --md5 /home/mastodon/live/public/system/media_attachments media/mastodata
./mc cp -r --md5 /home/mastodon/live/public/system/accounts media/mastodata
./mc cp -r --md5 /home/mastodon/live/public/system/cache media/mastodata


Double-check the bucket file structure in the MinIO admin console to ensure that the paths are correct (e.g. no duplicated accounts/accounts)

Configure Mastodon

Create a proxy in nginx using the instructions from joinmastodon.org

Matching the example links above, this is what you replace in the nginx configuration:

  • server_name mastodon-media.your-server.social;
  • set $s3_backend 'https: //media.your-server-social:9000/mastodata';
  • proxy_set_header Host media.your-server.social;
  • ssl_certificate /etc/letsencrypt/live/mastodon-media.your-server.social/fullchain.pem; # managed by Certbot
  • ssl_certificate_key /etc/letsencrypt/live/mastodon-media.your-server.social/privkey.pem; # managed by Certbot


Edit /etc/nginx/sites-available/mastodon (create a backup first, just in case!) and change the contents of the ~ ^/system/ location:

location ~ ^/system/ {
rewrite ^/system(.*) https: //mastodon-media.your-server.social/mastodata$1 permanent;
}


Edit /home/mastodon/live/.env.production and add the S3 configuration:

S3_ENABLED=true
S3_BUCKET=mastodata
AWS_ACCESS_KEY_ID=minio-mastodon-user-access-key-id
AWS_SECRET_ACCESS_KEY=minio-mastodon-user-access-key-secret
S3_REGION=minio-region
S3_ALIAS_HOST=mastodon-media.your.server.social
S3_HOSTNAME=media.your-server.social:9000
S3_ENDPOINT=https://media.your-server.social:9000
S3_PERMISSION=mastodon-readwrite

Restart the services:
sudo systemctl restart nginx
sudo systemctl restart mastodon-*


Now test your site. Right-click on the images and check they are being served from mastodon-media.your-server.social
  • Header on the "about page"
  • Custom Emojis
  • Profile headers & avatars
  • Preview cards
  • Existing media upload
  • New media upload
  • Media federate correctly to other servers


Visit the admin panel for any warnings about the S3 security configuration.

If media addresses are correct and you can see them in a new tab, but they are blank on your server, check the Content-Security-Policy headers in the ~ ^/system/ and @proxy locations in /etc/nginx/sites-available/mastodon.

Copy any new media files

Once everything is confirmed to be working, you are ready to copy any new files since you last copied them.

We use the mirror option to save time. You can optionally define a time interval too to skip older files:

./mc mirror --overwrite --newer-than 1d2h30m -a --md5 /home/mastodon/live/public/system/ media/mastodon/

To empty out the server, we can also use the move command, but make sure to create a backup of your server's media files, in case anything should go wrong (you can skip the cache). This command does not come with an --md5 option.

./mc mv -r /home/mastodon/live/public/system/site_uploads media/mastodata
./mc mv -r /home/mastodon/live/public/system/custom_emojis media/mastodata
./mc mv -r /home/mastodon/live/public/system/media_attachments media/mastodata
./mc mv -r /home/mastodon/live/public/system/accounts media/mastodata
./mc mv -r /home/mastodon/live/public/system/cache media/mastodata


This will still leave you with a tree of empty directories, so:
cd /home/mastodon/live/public/system
rm -rf accounts/
rm -rf cache/
rm -rf custom_emojis/
rm -rf media_attachments/
rm -rf site_uploads/


You might also have accumulated some orphans during the process, so:
su - mastodon
live/bin/tootctl media remove-orphans
exit


If you should be missing a remote status image here and there, you can restore them with the tootctl media refresh command.

And don't forget to reactivate your relays!

Automated backups

For setting up automated alerts and backups, follow the procedure in the Maintenance article.

I skip the cache to save disk space.

In the long run, it's probably best to set up a Multi-Drive MinIO.

rsync commands for settings & raw data:
  • rsync -av --delete-after -e ssh admin@media.server:/root/ /path/to/backup/media/root --exclude=".*"
  • rsync -av --delete-after -e ssh admin@media.server:/home/ /path/to/backup/media/home
  • rsync -av --delete-after -e ssh admin@media.server:/usr/lib/systemd/ /path/to/backup/media/usr/lib/systemd
  • rsync -av --delete-after -e ssh admin@media.server:/lib/systemd/system /path/to/backup/media/systemd
  • rsync -av --delete-after -e ssh admin@media.server:/etc/default/ /path/to/backup/media/etc/default
  • rsync -av --delete-after -e ssh admin@media.server:/var/spool/cron/crontabs /path/to/backup/media/
  • rsync -av --delete-after -e ssh admin@media.server:/storage/location/for/minio/ /storage/location/for/minio --exclude=minio/mastodata/cache --exclude=minio/.minio.sys/tmp


You can also use mc (see installation instructions above). Configure once with:
./mc alias set media-server https: //media.media-server.net:9000 mastodata password

Then run regularly:
  • /path/to/mc mirror --overwrite -a --md5 media-server/mastodata/accounts/ /path/to/backup/media/mastodata/accounts/
  • /path/to/mc mirror --overwrite -a --md5 media-server/mastodata/custom_emojis/ /path/to/backup/media/mastodata/custom_emojis/
  • /path/to/mc mirror --overwrite -a --md5 media-server/mastodata/site_uploads/ /path/to/backup/media/mastodata/site_uploads/
  • /path/to/mc mirror --overwrite -a --md5 media-server/mastodata/media_attachments/ /path/to/backup/media/mastodata/media_attachments/


My rsnapshot commands look like this:

  • backup_exec /path/to/mc mirror --overwrite --remove -a --md5 media-server/mastodata/accounts/ /path/to/backup/media/mastodata/accounts/
  • backup_exec /path/to/mc mirror --overwrite --remove -a --md5 media-server/mastodata/custom_emojis/ /path/to/backup/media/mastodata/custom_emojis/
  • backup_exec /path/to/mc mirror --overwrite --remove -a --md5 media-server/mastodata/site_uploads/ /path/to/backup/media/mastodata/site_uploads/
  • backup_exec /path/to/mc mirror --overwrite --remove -a --md5 media-server/mastodata/media_attachments/ /path/to/backup/media/mastodata/media_attachments/
  • backup /path/to/backup/media/mastodata media/


Last edited: 17 March 2024 07:32:26