From 7cd662f18c202ee2b099305e990d15a01370d6f9 Mon Sep 17 00:00:00 2001 From: Ivan Tashkinov Date: Thu, 24 Sep 2020 18:02:50 +0300 Subject: [PATCH 1/6] Added documentation on installation of `ffmpeg` and `ImageMagick` dependencies. --- CHANGELOG.md | 3 ++- docs/installation/alpine_linux_en.md | 9 +++++++++ docs/installation/arch_linux_en.md | 4 +++- docs/installation/debian_based_en.md | 4 +++- docs/installation/debian_based_jp.md | 4 +++- docs/installation/freebsd_en.md | 2 +- docs/installation/gentoo_en.md | 4 +++- docs/installation/netbsd_en.md | 2 +- docs/installation/openbsd_en.md | 5 +++-- docs/installation/openbsd_fi.md | 2 +- docs/installation/otp_en.md | 6 ++++-- 11 files changed, 33 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f5d01af3..3e5f8a504 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,9 +12,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false. - Users with the `discoverable` field set to false will not show up in searches. - Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option). +- **Breaking:** Introduced dependencies on `ffmpeg` and `ImageMagick` software packages. Please refer to documentation in `docs/installation`. ### Added -- Media preview proxy (requires media proxy be enabled; see `:media_preview_proxy` config for more details). +- Media preview proxy (requires `ffmpeg` and `ImageMagick` be installed and media proxy be enabled; see `:media_preview_proxy` config for more details). - Pleroma API: Importing the mutes users from CSV files. - Experimental websocket-based federation between Pleroma instances. - Admin API: Importing emoji from a zip file diff --git a/docs/installation/alpine_linux_en.md b/docs/installation/alpine_linux_en.md index a5683f18c..0b9f5cdae 100644 --- a/docs/installation/alpine_linux_en.md +++ b/docs/installation/alpine_linux_en.md @@ -13,6 +13,8 @@ It assumes that you have administrative rights, either as root or a user with [s * `erlang-parsetools` * `erlang-xmerl` * `git` +* `ffmpeg` +* `ImageMagick` * Development Tools * `cmake` @@ -56,6 +58,13 @@ sudo apk add erlang erlang-runtime-tools erlang-xmerl elixir ```shell sudo apk add erlang-eldap ``` + +### Install ffmpeg and ImageMagick + +```shell +sudo apk add ffmpeg imagemagick +``` + ### Install PostgreSQL * Install Postgresql server: diff --git a/docs/installation/arch_linux_en.md b/docs/installation/arch_linux_en.md index 7fb69dd60..38bb82432 100644 --- a/docs/installation/arch_linux_en.md +++ b/docs/installation/arch_linux_en.md @@ -10,6 +10,8 @@ This guide will assume that you have administrative rights, either as root or a * `git` * `base-devel` * `cmake` +* `ffmpeg` +* `ImageMagick` #### Optional packages used in this guide @@ -27,7 +29,7 @@ sudo pacman -Syu * Install some of the above mentioned programs: ```shell -sudo pacman -S git base-devel elixir cmake +sudo pacman -S git base-devel elixir cmake ffmpeg imagemagick ``` ### Install PostgreSQL diff --git a/docs/installation/debian_based_en.md b/docs/installation/debian_based_en.md index 60c2f47e5..30ee64c01 100644 --- a/docs/installation/debian_based_en.md +++ b/docs/installation/debian_based_en.md @@ -13,6 +13,8 @@ This guide will assume you are on Debian Stretch. This guide should also work wi * `git` * `build-essential` * `cmake` +* `ffmpeg` +* `ImageMagick` #### Optional packages used in this guide @@ -31,7 +33,7 @@ sudo apt full-upgrade * Install some of the above mentioned programs: ```shell -sudo apt install git build-essential postgresql postgresql-contrib cmake +sudo apt install git build-essential postgresql postgresql-contrib cmake ffmpeg imagemagick ``` ### Install Elixir and Erlang diff --git a/docs/installation/debian_based_jp.md b/docs/installation/debian_based_jp.md index c2dd840d3..219f40525 100644 --- a/docs/installation/debian_based_jp.md +++ b/docs/installation/debian_based_jp.md @@ -17,6 +17,8 @@ - `git` - `build-essential` - `cmake` +- `ffmpeg` +- `ImageMagick` #### このガイドで利用している追加パッケージ @@ -33,7 +35,7 @@ sudo apt full-upgrade * 上記に挙げたパッケージをインストールしておきます。 ``` -sudo apt install git build-essential postgresql postgresql-contrib cmake +sudo apt install git build-essential postgresql postgresql-contrib cmake ffmpeg imagemagick ``` diff --git a/docs/installation/freebsd_en.md b/docs/installation/freebsd_en.md index ca2575d9b..01b410a46 100644 --- a/docs/installation/freebsd_en.md +++ b/docs/installation/freebsd_en.md @@ -7,7 +7,7 @@ This document was written for FreeBSD 12.1, but should be work on future release This assumes the target system has `pkg(8)`. ``` -# pkg install elixir postgresql12-server postgresql12-client postgresql12-contrib git-lite sudo nginx gmake acme.sh cmake +# pkg install elixir postgresql12-server postgresql12-client postgresql12-contrib git-lite sudo nginx gmake acme.sh cmake ffmpeg imagemagick ``` Copy the rc.d scripts to the right directory: diff --git a/docs/installation/gentoo_en.md b/docs/installation/gentoo_en.md index 5a676380c..9a8b54ece 100644 --- a/docs/installation/gentoo_en.md +++ b/docs/installation/gentoo_en.md @@ -29,6 +29,8 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i * `dev-lang/elixir` * `dev-vcs/git` * `dev-util/cmake` +* `media-video/ffmpeg` +* `media-gfx/imagemagick` #### Optional ebuilds used in this guide @@ -47,7 +49,7 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i * Emerge all required the required and suggested software in one go: ```shell - # emerge --ask dev-db/postgresql dev-lang/elixir dev-vcs/git www-servers/nginx app-crypt/certbot app-crypt/certbot-nginx dev-util/cmake + # emerge --ask dev-db/postgresql dev-lang/elixir dev-vcs/git www-servers/nginx app-crypt/certbot app-crypt/certbot-nginx dev-util/cmake media-video/ffmpeg media-gfx/imagemagick ``` If you would not like to install the optional packages, remove them from this line. diff --git a/docs/installation/netbsd_en.md b/docs/installation/netbsd_en.md index 6ad0de2f6..8cf2e4682 100644 --- a/docs/installation/netbsd_en.md +++ b/docs/installation/netbsd_en.md @@ -10,7 +10,7 @@ Pleroma uses. The `mksh` shell is needed to run the Elixir `mix` script. -`# pkgin install acmesh elixir git-base git-docs mksh nginx postgresql11-server postgresql11-client postgresql11-contrib sudo` +`# pkgin install acmesh elixir git-base git-docs mksh nginx postgresql11-server postgresql11-client postgresql11-contrib sudo ffmpeg4 ImageMagick` You can also build these packages using pkgsrc: ``` diff --git a/docs/installation/openbsd_en.md b/docs/installation/openbsd_en.md index eee452845..277e5ec43 100644 --- a/docs/installation/openbsd_en.md +++ b/docs/installation/openbsd_en.md @@ -10,16 +10,17 @@ The following packages need to be installed: * elixir * gmake - * ImageMagick * git * postgresql-server * postgresql-contrib * cmake + * ffmpeg + * ImageMagick To install them, run the following command (with doas or as root): ``` -pkg_add elixir gmake ImageMagick git postgresql-server postgresql-contrib cmake +pkg_add elixir gmake git postgresql-server postgresql-contrib cmake ffmpeg ImageMagick ``` Pleroma requires a reverse proxy, OpenBSD has relayd in base (and is used in this guide) and packages/ports are available for nginx (www/nginx) and apache (www/apache-httpd). Independently of the reverse proxy, [acme-client(1)](https://man.openbsd.org/acme-client) can be used to get a certificate from Let's Encrypt. diff --git a/docs/installation/openbsd_fi.md b/docs/installation/openbsd_fi.md index b5b5056a9..2cb03e73e 100644 --- a/docs/installation/openbsd_fi.md +++ b/docs/installation/openbsd_fi.md @@ -16,7 +16,7 @@ Matrix-kanava #freenode_#pleroma:matrix.org ovat hyviä paikkoja löytää apua Asenna tarvittava ohjelmisto: -`# pkg_add git elixir gmake postgresql-server-10.3 postgresql-contrib-10.3 cmake` +`# pkg_add git elixir gmake postgresql-server-10.3 postgresql-contrib-10.3 cmake ffmpeg ImageMagick` Luo postgresql-tietokanta: diff --git a/docs/installation/otp_en.md b/docs/installation/otp_en.md index b7e3bb2ac..8fdf6b5b5 100644 --- a/docs/installation/otp_en.md +++ b/docs/installation/otp_en.md @@ -27,17 +27,19 @@ Other than things bundled in the OTP release Pleroma depends on: * PostgreSQL (also utilizes extensions in postgresql-contrib) * nginx (could be swapped with another reverse proxy but this guide covers only it) * certbot (for Let's Encrypt certificates, could be swapped with another ACME client, but this guide covers only it) +* ffmpeg (needed for media preview proxy) +* ImageMagick (needed for media preview proxy) === "Alpine" ``` echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories apk update - apk add curl unzip ncurses postgresql postgresql-contrib nginx certbot + apk add curl unzip ncurses postgresql postgresql-contrib nginx certbot ffmpeg imagemagick ``` === "Debian/Ubuntu" ``` - apt install curl unzip libncurses5 postgresql postgresql-contrib nginx certbot + apt install curl unzip libncurses5 postgresql postgresql-contrib nginx certbot ffmpeg imagemagick ``` ## Setup From 4e4f77108207157a49a627edb03951e2f15b62f1 Mon Sep 17 00:00:00 2001 From: Ivan Tashkinov Date: Sat, 26 Sep 2020 19:32:16 +0300 Subject: [PATCH 2/6] Adjusted MediaProxyControllerTest to gracefully fail on missing dependencies. Installation docs update. Added ffmpeg/imagemagick checks to launch checks (if media preview proxy is enabled). Added documentation on installing optional media / graphics packages (imagemagick, ffmpeg, exiftool). --- docs/installation/alpine_linux_en.md | 18 ++++++------ docs/installation/arch_linux_en.md | 13 +++++++-- docs/installation/debian_based_en.md | 13 +++++++-- docs/installation/debian_based_jp.md | 12 ++++++-- docs/installation/freebsd_en.md | 8 +++++- docs/installation/gentoo_en.md | 13 +++++++-- docs/installation/netbsd_en.md | 4 +++ docs/installation/openbsd_en.md | 13 +++++++++ docs/installation/openbsd_fi.md | 11 ++++++++ .../optional/media_graphics_packages.md | 32 ++++++++++++++++++++++ docs/installation/otp_en.md | 25 ++++++++++++++--- lib/pleroma/application.ex | 18 +++++++++++- lib/pleroma/helpers/media_helper.ex | 12 ++++++++ .../media_proxy/media_proxy_controller_test.exs | 15 ++++++++++ 14 files changed, 180 insertions(+), 27 deletions(-) create mode 100644 docs/installation/optional/media_graphics_packages.md diff --git a/docs/installation/alpine_linux_en.md b/docs/installation/alpine_linux_en.md index 0b9f5cdae..d89c7f46f 100644 --- a/docs/installation/alpine_linux_en.md +++ b/docs/installation/alpine_linux_en.md @@ -13,8 +13,6 @@ It assumes that you have administrative rights, either as root or a user with [s * `erlang-parsetools` * `erlang-xmerl` * `git` -* `ffmpeg` -* `ImageMagick` * Development Tools * `cmake` @@ -22,6 +20,9 @@ It assumes that you have administrative rights, either as root or a user with [s * `nginx` (preferred, example configs for other reverse proxies can be found in the repo) * `certbot` (or any other ACME client for Let’s Encrypt certificates) +* `ImageMagick` +* `ffmpeg` +* `exiftool` ### Prepare the system @@ -31,7 +32,6 @@ It assumes that you have administrative rights, either as root or a user with [s awk 'NR==2' /etc/apk/repositories | sed 's/main/community/' | tee -a /etc/apk/repositories ``` - * Then update the system, if not already done: ```shell @@ -59,12 +59,6 @@ sudo apk add erlang erlang-runtime-tools erlang-xmerl elixir sudo apk add erlang-eldap ``` -### Install ffmpeg and ImageMagick - -```shell -sudo apk add ffmpeg imagemagick -``` - ### Install PostgreSQL * Install Postgresql server: @@ -85,6 +79,12 @@ sudo /etc/init.d/postgresql start sudo rc-update add postgresql ``` +### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)) + +```shell +sudo apk add ffmpeg imagemagick exiftool +``` + ### Install PleromaBE * Add a new system user for the Pleroma service: diff --git a/docs/installation/arch_linux_en.md b/docs/installation/arch_linux_en.md index 38bb82432..724b4660a 100644 --- a/docs/installation/arch_linux_en.md +++ b/docs/installation/arch_linux_en.md @@ -10,13 +10,14 @@ This guide will assume that you have administrative rights, either as root or a * `git` * `base-devel` * `cmake` -* `ffmpeg` -* `ImageMagick` #### Optional packages used in this guide * `nginx` (preferred, example configs for other reverse proxies can be found in the repo) * `certbot` (or any other ACME client for Let’s Encrypt certificates) +* `ImageMagick` +* `ffmpeg` +* `exiftool` ### Prepare the system @@ -29,7 +30,7 @@ sudo pacman -Syu * Install some of the above mentioned programs: ```shell -sudo pacman -S git base-devel elixir cmake ffmpeg imagemagick +sudo pacman -S git base-devel elixir cmake ``` ### Install PostgreSQL @@ -54,6 +55,12 @@ sudo -iu postgres initdb -D /var/lib/postgres/data sudo systemctl enable --now postgresql.service ``` +### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)) + +```shell +sudo pacman -S ffmpeg imagemagick perl-image-exiftool +``` + ### Install PleromaBE * Add a new system user for the Pleroma service: diff --git a/docs/installation/debian_based_en.md b/docs/installation/debian_based_en.md index 30ee64c01..eac499a29 100644 --- a/docs/installation/debian_based_en.md +++ b/docs/installation/debian_based_en.md @@ -13,13 +13,14 @@ This guide will assume you are on Debian Stretch. This guide should also work wi * `git` * `build-essential` * `cmake` -* `ffmpeg` -* `ImageMagick` #### Optional packages used in this guide * `nginx` (preferred, example configs for other reverse proxies can be found in the repo) * `certbot` (or any other ACME client for Let’s Encrypt certificates) +* `ImageMagick` +* `ffmpeg` +* `exiftool` ### Prepare the system @@ -33,7 +34,7 @@ sudo apt full-upgrade * Install some of the above mentioned programs: ```shell -sudo apt install git build-essential postgresql postgresql-contrib cmake ffmpeg imagemagick +sudo apt install git build-essential postgresql postgresql-contrib cmake ``` ### Install Elixir and Erlang @@ -52,6 +53,12 @@ sudo apt update sudo apt install elixir erlang-dev erlang-nox ``` +### Optional packages: [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md) + +```shell +sudo apt install imagemagick ffmpeg libimage-exiftool-perl +``` + ### Install PleromaBE * Add a new system user for the Pleroma service: diff --git a/docs/installation/debian_based_jp.md b/docs/installation/debian_based_jp.md index 219f40525..764afbe1a 100644 --- a/docs/installation/debian_based_jp.md +++ b/docs/installation/debian_based_jp.md @@ -17,13 +17,14 @@ - `git` - `build-essential` - `cmake` -- `ffmpeg` -- `ImageMagick` #### このガイドで利用している追加パッケージ - `nginx` (おすすめです。他のリバースプロキシを使う場合は、参考となる設定をこのリポジトリから探してください) - `certbot` (または何らかのLet's Encrypt向けACMEクライアント) +- `ImageMagick` +- `ffmpeg` +- `exiftool` ### システムを準備する @@ -38,7 +39,6 @@ sudo apt full-upgrade sudo apt install git build-essential postgresql postgresql-contrib cmake ffmpeg imagemagick ``` - ### ElixirとErlangをインストールします * Erlangのリポジトリをダウンロードおよびインストールします。 @@ -53,6 +53,12 @@ sudo apt update sudo apt install elixir erlang-dev erlang-nox ``` +### オプションパッケージ: [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md) + +```shell +sudo apt install imagemagick ffmpeg libimage-exiftool-perl +``` + ### Pleroma BE (バックエンド) をインストールします * Pleroma用に新しいユーザーを作ります。 diff --git a/docs/installation/freebsd_en.md b/docs/installation/freebsd_en.md index 01b410a46..fdcb06c53 100644 --- a/docs/installation/freebsd_en.md +++ b/docs/installation/freebsd_en.md @@ -7,7 +7,7 @@ This document was written for FreeBSD 12.1, but should be work on future release This assumes the target system has `pkg(8)`. ``` -# pkg install elixir postgresql12-server postgresql12-client postgresql12-contrib git-lite sudo nginx gmake acme.sh cmake ffmpeg imagemagick +# pkg install elixir postgresql12-server postgresql12-client postgresql12-contrib git-lite sudo nginx gmake acme.sh cmake ``` Copy the rc.d scripts to the right directory: @@ -26,6 +26,12 @@ Setup the required services to automatically start at boot, using `sysrc(8)`. # service postgresql start ``` +### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)) + +```shell +# pkg install imagemagick ffmpeg p5-Image-ExifTool +``` + ## Configuring Pleroma Create a user for Pleroma: diff --git a/docs/installation/gentoo_en.md b/docs/installation/gentoo_en.md index 9a8b54ece..638fc4e47 100644 --- a/docs/installation/gentoo_en.md +++ b/docs/installation/gentoo_en.md @@ -29,14 +29,15 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i * `dev-lang/elixir` * `dev-vcs/git` * `dev-util/cmake` -* `media-video/ffmpeg` -* `media-gfx/imagemagick` #### Optional ebuilds used in this guide * `www-servers/nginx` (preferred, example configs for other reverse proxies can be found in the repo) * `app-crypt/certbot` (or any other ACME client for Let’s Encrypt certificates) * `app-crypt/certbot-nginx` (nginx certbot plugin that allows use of the all-powerful `--nginx` flag on certbot) +* `media-gfx/imagemagick` +* `media-video/ffmpeg` +* `media-libs/exiftool` ### Prepare the system @@ -49,7 +50,7 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i * Emerge all required the required and suggested software in one go: ```shell - # emerge --ask dev-db/postgresql dev-lang/elixir dev-vcs/git www-servers/nginx app-crypt/certbot app-crypt/certbot-nginx dev-util/cmake media-video/ffmpeg media-gfx/imagemagick + # emerge --ask dev-db/postgresql dev-lang/elixir dev-vcs/git www-servers/nginx app-crypt/certbot app-crypt/certbot-nginx dev-util/cmake ``` If you would not like to install the optional packages, remove them from this line. @@ -89,6 +90,12 @@ If you do not plan to make any modifications to your Pleroma instance, cloning d Not only does this make it much easier to deploy changes you make, as you can commit and pull from upstream and all that good stuff from the comfort of your local machine then simply `git pull` on your instance server when you're ready to deploy, it also ensures you are compliant with the Affero General Public Licence that Pleroma is licenced under, which stipulates that all network services provided with modified AGPL code must publish their changes on a publicly available internet service and for free. It also makes it much easier to ask for help from and provide help to your fellow Pleroma admins if your public repo always reflects what you are running because it is part of your deployment procedure. +### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)) + +```shell +# emerge --ask media-video/ffmpeg media-gfx/imagemagick media-libs/exiftool +``` + ### Install PleromaBE * Add a new system user for the Pleroma service and set up default directories: diff --git a/docs/installation/netbsd_en.md b/docs/installation/netbsd_en.md index 8cf2e4682..d5fa04fdf 100644 --- a/docs/installation/netbsd_en.md +++ b/docs/installation/netbsd_en.md @@ -44,6 +44,10 @@ pgsql=YES First, run `# /etc/rc.d/pgsql start`. Then, `$ sudo -Hu pgsql -g pgsql createdb`. +### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)) + +`# pkgin install ImageMagick ffmpeg4 p5-Image-ExifTool` + ## Configuring Pleroma Create a user for Pleroma: diff --git a/docs/installation/openbsd_en.md b/docs/installation/openbsd_en.md index 277e5ec43..8092ac379 100644 --- a/docs/installation/openbsd_en.md +++ b/docs/installation/openbsd_en.md @@ -25,6 +25,19 @@ pkg_add elixir gmake git postgresql-server postgresql-contrib cmake ffmpeg Image Pleroma requires a reverse proxy, OpenBSD has relayd in base (and is used in this guide) and packages/ports are available for nginx (www/nginx) and apache (www/apache-httpd). Independently of the reverse proxy, [acme-client(1)](https://man.openbsd.org/acme-client) can be used to get a certificate from Let's Encrypt. +#### Optional software + +Per [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md): + * ImageMagick + * ffmpeg + * exiftool + +To install the above: + +``` +pkg_add ImageMagick ffmpeg p5-Image-ExifTool +``` + #### Creating the pleroma user Pleroma will be run by a dedicated user, \_pleroma. Before creating it, insert the following lines in login.conf: ``` diff --git a/docs/installation/openbsd_fi.md b/docs/installation/openbsd_fi.md index 2cb03e73e..01cf34ab4 100644 --- a/docs/installation/openbsd_fi.md +++ b/docs/installation/openbsd_fi.md @@ -18,6 +18,17 @@ Asenna tarvittava ohjelmisto: `# pkg_add git elixir gmake postgresql-server-10.3 postgresql-contrib-10.3 cmake ffmpeg ImageMagick` +#### Optional software + +[`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md): + * ImageMagick + * ffmpeg + * exiftool + +Asenna tarvittava ohjelmisto: + +`# pkg_add ImageMagick ffmpeg p5-Image-ExifTool` + Luo postgresql-tietokanta: `# su - _postgresql` diff --git a/docs/installation/optional/media_graphics_packages.md b/docs/installation/optional/media_graphics_packages.md new file mode 100644 index 000000000..cb3d71188 --- /dev/null +++ b/docs/installation/optional/media_graphics_packages.md @@ -0,0 +1,32 @@ +# Optional software packages needed for specific functionality + +For specific Pleroma functionality (which is disabled by default) some or all of the below packages are required: + * `ImageMagic` + * `ffmpeg` + * `exiftool` + +Please refer to documentation in `docs/installation` on how to install them on specific OS. + +Note: the packages are not required with the current default settings of Pleroma. + +## `ImageMagick` + +`ImageMagick` is a set of tools to create, edit, compose, or convert bitmap images. + +It is required for the following Pleroma features: + * `Pleroma.Upload.Filters.Mogrify`, `Pleroma.Upload.Filters.Mogrifun` upload filters (related config: `Plaroma.Upload/filters` in `config/config.exs`) + * Media preview proxy for still images (related config: `media_preview_proxy/enabled` in `config/config.exs`) + +## `ffmpeg` + +`ffmpeg` is software to record, convert and stream audio and video. + +It is required for the following Pleroma features: + * Media preview proxy for videos (related config: `media_preview_proxy/enabled` in `config/config.exs`) + +## `exiftool` + +`exiftool` is media files metadata reader/writer. + +It is required for the following Pleroma features: + * `Pleroma.Upload.Filters.Exiftool` upload filter (related config: `Plaroma.Upload/filters` in `config/config.exs`) diff --git a/docs/installation/otp_en.md b/docs/installation/otp_en.md index 8fdf6b5b5..f6b05c887 100644 --- a/docs/installation/otp_en.md +++ b/docs/installation/otp_en.md @@ -27,19 +27,36 @@ Other than things bundled in the OTP release Pleroma depends on: * PostgreSQL (also utilizes extensions in postgresql-contrib) * nginx (could be swapped with another reverse proxy but this guide covers only it) * certbot (for Let's Encrypt certificates, could be swapped with another ACME client, but this guide covers only it) -* ffmpeg (needed for media preview proxy) -* ImageMagick (needed for media preview proxy) === "Alpine" ``` echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories apk update - apk add curl unzip ncurses postgresql postgresql-contrib nginx certbot ffmpeg imagemagick + apk add curl unzip ncurses postgresql postgresql-contrib nginx certbot ``` === "Debian/Ubuntu" ``` - apt install curl unzip libncurses5 postgresql postgresql-contrib nginx certbot ffmpeg imagemagick + apt install curl unzip libncurses5 postgresql postgresql-contrib nginx certbot + ``` + +### Installing optional packages + +Per [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md): + * ImageMagick + * ffmpeg + * exiftool + +=== "Alpine" + ``` + echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories + apk update + apk add imagemagick ffmpeg exiftool + ``` + +=== "Debian/Ubuntu" + ``` + apt install imagemagick ffmpeg libimage-exiftool-perl ``` ## Setup diff --git a/lib/pleroma/application.ex b/lib/pleroma/application.ex index 00ec79a2a..d7d8e423e 100644 --- a/lib/pleroma/application.ex +++ b/lib/pleroma/application.ex @@ -268,7 +268,8 @@ defmodule Pleroma.Application do with true <- filter in filters, false <- Pleroma.Utils.command_available?(command_required) do Logger.error( - "#{filter} is specified in list of Pleroma.Upload filters, but the #{command_required} command is not found" + "#{filter} is specified in list of Pleroma.Upload filters, but the " <> + "#{command_required} command is not found" ) end end @@ -276,5 +277,20 @@ defmodule Pleroma.Application do check_filter.(Pleroma.Upload.Filters.Exiftool, "exiftool") check_filter.(Pleroma.Upload.Filters.Mogrify, "mogrify") check_filter.(Pleroma.Upload.Filters.Mogrifun, "mogrify") + + with true <- Config.get([:media_preview_proxy, :enabled]), + missing_graphics_tools = Pleroma.Helpers.MediaHelper.missing_dependencies(), + [] <- missing_graphics_tools do + :noop + else + false -> + :noop + + missing_graphics_tools -> + Logger.error( + "The following dependencies required by Media preview proxy " <> + "(which is currently enabled) are not installed: #{inspect(missing_graphics_tools)}" + ) + end end end diff --git a/lib/pleroma/helpers/media_helper.ex b/lib/pleroma/helpers/media_helper.ex index b6f35a24b..6b799173e 100644 --- a/lib/pleroma/helpers/media_helper.ex +++ b/lib/pleroma/helpers/media_helper.ex @@ -9,6 +9,18 @@ defmodule Pleroma.Helpers.MediaHelper do alias Pleroma.HTTP + require Logger + + def missing_dependencies do + Enum.reduce([imagemagick: "convert", ffmpeg: "ffmpeg"], [], fn {sym, executable}, acc -> + if Pleroma.Utils.command_available?(executable) do + acc + else + [sym | acc] + end + end) + end + def image_resize(url, options) do with executable when is_binary(executable) <- System.find_executable("convert"), {:ok, args} <- prepare_image_resize_args(options), diff --git a/test/web/media_proxy/media_proxy_controller_test.exs b/test/web/media_proxy/media_proxy_controller_test.exs index 33e6873f7..e9b584822 100644 --- a/test/web/media_proxy/media_proxy_controller_test.exs +++ b/test/web/media_proxy/media_proxy_controller_test.exs @@ -81,6 +81,15 @@ defmodule Pleroma.Web.MediaProxy.MediaProxyControllerTest do end describe "Media Preview Proxy" do + def assert_dependencies_installed do + missing_dependencies = Pleroma.Helpers.MediaHelper.missing_dependencies() + + assert missing_dependencies == [], + "Error: missing dependencies (please refer to `docs/installation`): #{ + inspect(missing_dependencies) + }" + end + setup do clear_config([:media_proxy, :enabled], true) clear_config([:media_preview_proxy, :enabled], true) @@ -184,6 +193,8 @@ defmodule Pleroma.Web.MediaProxy.MediaProxyControllerTest do url: url, media_proxy_url: media_proxy_url } do + assert_dependencies_installed() + # Setting a high :min_content_length to ensure this scenario is not affected by its logic clear_config([:media_preview_proxy, :min_content_length], 1_000_000_000) @@ -270,6 +281,8 @@ defmodule Pleroma.Web.MediaProxy.MediaProxyControllerTest do url: url, media_proxy_url: media_proxy_url } do + assert_dependencies_installed() + Tesla.Mock.mock(fn %{method: "head", url: ^media_proxy_url} -> %Tesla.Env{status: 200, body: "", headers: [{"content-type", "image/png"}]} @@ -290,6 +303,8 @@ defmodule Pleroma.Web.MediaProxy.MediaProxyControllerTest do url: url, media_proxy_url: media_proxy_url } do + assert_dependencies_installed() + Tesla.Mock.mock(fn %{method: "head", url: ^media_proxy_url} -> %Tesla.Env{status: 200, body: "", headers: [{"content-type", "image/jpeg"}]} From ec2f70cd23ff77a1580650dd629a84f61df1d486 Mon Sep 17 00:00:00 2001 From: lain Date: Mon, 28 Sep 2020 14:14:32 +0200 Subject: [PATCH 3/6] CI: Add ffmpeg --- .gitlab-ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index dc953a929..121e4abfe 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -59,7 +59,7 @@ unit-testing: alias: postgres command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"] script: - - apt-get update && apt-get install -y libimage-exiftool-perl + - apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg - mix deps.get - mix ecto.create - mix ecto.migrate @@ -93,7 +93,7 @@ unit-testing-rum: <<: *global_variables RUM_ENABLED: "true" script: - - apt-get update && apt-get install -y libimage-exiftool-perl + - apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg - mix deps.get - mix ecto.create - mix ecto.migrate From 7d5c3883acafc2c84f65f38dd639d4999f14215a Mon Sep 17 00:00:00 2001 From: Ivan Tashkinov Date: Tue, 29 Sep 2020 16:28:06 +0300 Subject: [PATCH 4/6] [#3031] Refactoring: moved system commands checks to ApplicationRequirements. --- lib/pleroma/application.ex | 34 ---------------- lib/pleroma/application_requirements.ex | 72 ++++++++++++++++++++++++++++++--- 2 files changed, 67 insertions(+), 39 deletions(-) diff --git a/lib/pleroma/application.ex b/lib/pleroma/application.ex index d7d8e423e..e73d89350 100644 --- a/lib/pleroma/application.ex +++ b/lib/pleroma/application.ex @@ -56,7 +56,6 @@ defmodule Pleroma.Application do Pleroma.ApplicationRequirements.verify!() setup_instrumenters() load_custom_modules() - check_system_commands() Pleroma.Docs.JSON.compile() adapter = Application.get_env(:tesla, :adapter) @@ -260,37 +259,4 @@ defmodule Pleroma.Application do end defp http_children(_, _), do: [] - - defp check_system_commands do - filters = Config.get([Pleroma.Upload, :filters]) - - check_filter = fn filter, command_required -> - with true <- filter in filters, - false <- Pleroma.Utils.command_available?(command_required) do - Logger.error( - "#{filter} is specified in list of Pleroma.Upload filters, but the " <> - "#{command_required} command is not found" - ) - end - end - - check_filter.(Pleroma.Upload.Filters.Exiftool, "exiftool") - check_filter.(Pleroma.Upload.Filters.Mogrify, "mogrify") - check_filter.(Pleroma.Upload.Filters.Mogrifun, "mogrify") - - with true <- Config.get([:media_preview_proxy, :enabled]), - missing_graphics_tools = Pleroma.Helpers.MediaHelper.missing_dependencies(), - [] <- missing_graphics_tools do - :noop - else - false -> - :noop - - missing_graphics_tools -> - Logger.error( - "The following dependencies required by Media preview proxy " <> - "(which is currently enabled) are not installed: #{inspect(missing_graphics_tools)}" - ) - end - end end diff --git a/lib/pleroma/application_requirements.ex b/lib/pleroma/application_requirements.ex index 16f62b6f5..b977257a3 100644 --- a/lib/pleroma/application_requirements.ex +++ b/lib/pleroma/application_requirements.ex @@ -9,6 +9,9 @@ defmodule Pleroma.ApplicationRequirements do defmodule VerifyError, do: defexception([:message]) + alias Pleroma.Config + alias Pleroma.Helpers.MediaHelper + import Ecto.Query require Logger @@ -16,7 +19,8 @@ defmodule Pleroma.ApplicationRequirements do @spec verify!() :: :ok | VerifyError.t() def verify! do :ok - |> check_confirmation_accounts! + |> check_system_commands!() + |> check_confirmation_accounts!() |> check_migrations_applied!() |> check_welcome_message_config!() |> check_rum!() @@ -48,7 +52,9 @@ defmodule Pleroma.ApplicationRequirements do if Pleroma.Config.get([:instance, :account_activation_required]) && not Pleroma.Config.get([Pleroma.Emails.Mailer, :enabled]) do Logger.error( - "Account activation enabled, but no Mailer settings enabled.\nPlease set config :pleroma, :instance, account_activation_required: false\nOtherwise setup and enable Mailer." + "Account activation enabled, but no Mailer settings enabled.\n" <> + "Please set config :pleroma, :instance, account_activation_required: false\n" <> + "Otherwise setup and enable Mailer." ) {:error, @@ -81,7 +87,9 @@ defmodule Pleroma.ApplicationRequirements do Enum.map(down_migrations, fn {:down, id, name} -> "- #{name} (#{id})\n" end) Logger.error( - "The following migrations were not applied:\n#{down_migrations_text}If you want to start Pleroma anyway, set\nconfig :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true" + "The following migrations were not applied:\n#{down_migrations_text}" <> + "If you want to start Pleroma anyway, set\n" <> + "config :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true" ) {:error, "Unapplied Migrations detected"} @@ -124,14 +132,22 @@ defmodule Pleroma.ApplicationRequirements do case {setting, migrate} do {true, false} -> Logger.error( - "Use `RUM` index is enabled, but were not applied migrations for it.\nIf you want to start Pleroma anyway, set\nconfig :pleroma, :database, rum_enabled: false\nOtherwise apply the following migrations:\n`mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/`" + "Use `RUM` index is enabled, but were not applied migrations for it.\n" <> + "If you want to start Pleroma anyway, set\n" <> + "config :pleroma, :database, rum_enabled: false\n" <> + "Otherwise apply the following migrations:\n" <> + "`mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/`" ) {:error, "Unapplied RUM Migrations detected"} {false, true} -> Logger.error( - "Detected applied migrations to use `RUM` index, but `RUM` isn't enable in settings.\nIf you want to use `RUM`, set\nconfig :pleroma, :database, rum_enabled: true\nOtherwise roll `RUM` migrations back.\n`mix ecto.rollback --migrations-path priv/repo/optional_migrations/rum_indexing/`" + "Detected applied migrations to use `RUM` index, but `RUM` isn't enable in settings.\n" <> + "If you want to use `RUM`, set\n" <> + "config :pleroma, :database, rum_enabled: true\n" <> + "Otherwise roll `RUM` migrations back.\n" <> + "`mix ecto.rollback --migrations-path priv/repo/optional_migrations/rum_indexing/`" ) {:error, "RUM Migrations detected"} @@ -140,4 +156,50 @@ defmodule Pleroma.ApplicationRequirements do :ok end end + + defp check_system_commands!(:ok) do + filter_commands_statuses = [ + check_filter(Pleroma.Upload.Filters.Exiftool, "exiftool"), + check_filter(Pleroma.Upload.Filters.Mogrify, "mogrify"), + check_filter(Pleroma.Upload.Filters.Mogrifun, "mogrify") + ] + + preview_proxy_commands_status = + if !Config.get([:media_preview_proxy, :enabled]) or + MediaHelper.missing_dependencies() == [] do + true + else + Logger.error( + "The following dependencies required by Media preview proxy " <> + "(which is currently enabled) are not installed: " <> + inspect(MediaHelper.missing_dependencies()) + ) + + false + end + + if Enum.all?([preview_proxy_commands_status | filter_commands_statuses], & &1) do + :ok + else + {:error, + "System commands missing. Check logs and see `docs/installation` for more details."} + end + end + + defp check_system_commands!(result), do: result + + defp check_filter(filter, command_required) do + filters = Config.get([Pleroma.Upload, :filters]) + + if filter in filters and not Pleroma.Utils.command_available?(command_required) do + Logger.error( + "#{filter} is specified in list of Pleroma.Upload filters, but the " <> + "#{command_required} command is not found" + ) + + false + else + true + end + end end From a24f80badf2c24a12de1e99a429aa23e2b40b40c Mon Sep 17 00:00:00 2001 From: lain Date: Tue, 29 Sep 2020 14:30:18 +0000 Subject: [PATCH 5/6] Apply 1 suggestion(s) to 1 file(s) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e5f8a504..8b0f72eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - **Breaking:** Introduced dependencies on `ffmpeg` and `ImageMagick` software packages. Please refer to documentation in `docs/installation`. ### Added -- Media preview proxy (requires `ffmpeg` and `ImageMagick` be installed and media proxy be enabled; see `:media_preview_proxy` config for more details). +- Media preview proxy (requires `ffmpeg` and `ImageMagick` to be installed and media proxy to be enabled; see `:media_preview_proxy` config for more details). - Pleroma API: Importing the mutes users from CSV files. - Experimental websocket-based federation between Pleroma instances. - Admin API: Importing emoji from a zip file From ae262846389aabfc71c12c2eb903cbe367140184 Mon Sep 17 00:00:00 2001 From: Ivan Tashkinov Date: Tue, 29 Sep 2020 17:34:49 +0300 Subject: [PATCH 6/6] [#3031] Adjusted changelog entry. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b0f72eb3..f11585113 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false. - Users with the `discoverable` field set to false will not show up in searches. - Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option). -- **Breaking:** Introduced dependencies on `ffmpeg` and `ImageMagick` software packages. Please refer to documentation in `docs/installation`. +- Introduced optional dependencies on `ffmpeg`, `ImageMagick`, `exiftool` software packages. Please refer to `docs/installation/optional/media_graphics_packages.md`. ### Added - Media preview proxy (requires `ffmpeg` and `ImageMagick` to be installed and media proxy to be enabled; see `:media_preview_proxy` config for more details).