Add Mastodon package and module #78810

happy-river · 2020-01-30T05:17:19Z

Motivation for this change

This PR continues the work done in #60788 by @petabyteboy, and adds these improvements:

Add a shell script to set up the environment variables for tootctl.
Make database setup work correctly with Mastodon 3.0.1.
Allow database and SMTP passwords to contain spaces.
Improve descriptions of module options.
Make pkgs.mastodon.override { ... } work to build forks of Mastodon.
Add a package option to the service module.
Borrow a fix from vagrant: Resolve crash by replacing gem symlinks with directories #76765 to make the package work with rubygems 3.1.2.
Remove the import <nixpkgs> from the source expression.
Make the update script work, and add a derivation for it.
Add documentation.

Things that could still be improved:

Investigate if it makes sense to allow building, enabling and disabling the three services (web, sidekiq and streaming) separately, similar to the kubernetes module.
Add more advanced options, i.e. the number of sidekiq threads, support for S3 storage backend, ...

Things done

ghost

Beautiful!
Only thing I noticed is the package tests fail.

Failures:

  1) Auth::RegistrationsController POST #create approval-based registrations with valid invite creates user
     Failure/Error: expect(user.locale).to eq(accept_language)

       expected: "sr-Latn"
            got: "sr"

       (compared using ==)
     # ./spec/controllers/auth/registrations_controller_spec.rb:197:in `block (4 levels) in <top (required)>'
     # ./spec/controllers/auth/registrations_controller_spec.rb:177:in `block (4 levels) in <top (required)>'
     # ./spec/controllers/auth/registrations_controller_spec.rb:9:in `block (3 levels) in <top (required)>'
     # ./spec/controllers/auth/registrations_controller_spec.rb:87:in `block (3 levels) in <top (required)>'

Finished in 8 minutes 47 seconds (files took 34.99 seconds to load)
2430 examples, 1 failure, 9 pending

Failed examples:

rspec ./spec/controllers/auth/registrations_controller_spec.rb:193 # Auth::RegistrationsController POST #create approval-based registrations with valid invite creates user

Randomized with seed 15353


error: command `su - tester -c 'cd mastodon; rspec --no-color'` failed (exit code 1)

Oh and Aarch64 build is failing, I suggest you just remove it from the supported architectures for now.

LoadError: /nix/store/wisk11g6scq2cbsi5ifxx10lq19j47il-mastodon-gems-3.0.1/lib/ruby/gems/2.6.0/gems/stackprof-0.2.12/lib/stackprof/stackprof.so: undefined symbol: pthread_atfork - /nix/store/wisk11g6scq2cbsi5ifxx10lq19j47il-mastodon-gems-3.0.1/lib/ruby/gems/2.6.0/gems/stackprof-0.2.12/lib/stackprof/stackprof.so

nixos/modules/services/web-apps/mastodon.nix

Mic92 · 2020-01-30T09:56:30Z

cc @schmittlauch

Mic92 · 2020-01-30T09:58:40Z

Hey. This looks good to me. Is it feasible though to add mastodons npm packages to
our node-packages set though? We are already doing this for other packages like lumo or iosevka:

nixpkgs/pkgs/development/node-packages/node-packages-v10.json

Line 80 in 8539d5f

, {"lumo-build-deps": "../interpreters/clojurescript/lumo" }

ghost · 2020-01-30T10:10:09Z

Hey. This looks good to me. Is it feasible though to add mastodons npm packages to
our node-packages set though? We are already doing this for other packages like lumo or iosevka:

nixpkgs/pkgs/development/node-packages/node-packages-v10.json

Line 80 in 8539d5f

, {"lumo-build-deps": "../interpreters/clojurescript/lumo" }

Afaik there are no official user-targeted mastodon npm packages. The webapp itself is not an npm package either, but it is exposed as a nix derivation in the nixpkgs attrset (as pkgs.mastodon.mastodon-assets). The yarn/npm dependencies used to build the client are exposed at pkgs.mastodon.mastodon-js-modules.
What would be the use case for exposing the build deps, which are not user-targeted, in node-packages?

Mic92 · 2020-01-30T11:05:36Z

@petabyteboy The idea is that we can share npm package source between different packages in nixpkgs instead of having 11.000 lines of generated nix expressions per package, which affects also evaluation time.

ghost · 2020-01-30T11:14:30Z

@petabyteboy The idea is that we can share npm package source between different packages in nixpkgs instead of having 11.000 lines of generated nix expressions per package, which affects also evaluation time.

Ah I see. But I doubt this would be possible since mastodon is using yarn and not npm.
It would be nice to see support for this kind of optimization in yarn2nix, possibly also sharing the node packages expression with node2nix.

Mic92 · 2020-01-30T11:23:12Z

Ah I see. But I doubt this would be possible since mastodon is using yarn and not npm.
It would be nice to see support for this kind of optimization in yarn2nix, possibly also sharing the node packages expression with node2nix.

Do both not use a packages.json. It thought that this could be read by npm as well.

ghost · 2020-01-30T12:30:50Z

Do both not use a packages.json. It thought that this could be read by npm as well.

Yes and no, the mastodon repository contains a yarn.lock file with extra version information that can not be described in the original package.json format. NPM added a package-lock.json file to support the same thing, but it is incompatible with the yarn.lock format.

Mic92 · 2020-01-30T14:14:30Z

Do both not use a packages.json. It thought that this could be read by npm as well.

Yes and no, the mastodon repository contains a yarn.lock file with extra version information that can not be described in the original package.json format. NPM added a package-lock.json file to support the same thing, but it is incompatible with the yarn.lock format.

So it won't capture the some indirect dependencies pinnings? Does this matter in our case though?

ghost · 2020-01-30T15:10:46Z

So it won't capture the some indirect dependencies pinnings? Does this matter in our case though?

I think so. Even if it is possible to package the dependencies with node2nix, the dependency versions could change every time the nix expression is generated. With such a complex application I think it is crucial to reproduce the dependencies as close to what upstream intends, and they clearly instruct users to use yarn when building the frontend.
I think it's more important to make sure that the application will work correctly before starting to optimize the evaluation time. Also it wouldn't be indirect dependencies only that are affected.

Mic92 · 2020-01-30T15:35:42Z

But on the other hand we also need to keep nixpkgs usuable. This packages adds 435.2KB of data to nixpkgs that everyone needs to download (Our node-packages.nix is 2.6M with all our packages). I get regularly complaints of people not being able to run nixpkgs-review with local evaluation because lack of RAM.

happy-river · 2020-01-31T01:17:30Z

   expected: "sr-Latn"
        got: "sr"

Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?

ghost · 2020-01-31T09:23:09Z

   expected: "sr-Latn"
        got: "sr"
Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?

No, this output is from the NixOS test you wrote.

happy-river · 2020-01-31T15:52:37Z

   expected: "sr-Latn"
        got: "sr"
Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?
No, this output is from the NixOS test you wrote.

On further investigation, it appears to be a bug in the Mastodon test which happens rarely due to the use of randomization in the test. I'll report it upstream.

nyanloutre · 2020-02-01T15:52:27Z

After some testing I figured there should be an option to disable SMTP_LOGIN and SMTP_PASSWORD as it will block access to local smtp server without authentication.
Even with auth method to none like in the following example it will not work if the login and password variables are defined:

SMTP_AUTH_METHOD = "none";
SMTP_OPENSSL_VERIFY_MODE = "none";

happy-river · 2020-02-08T15:56:12Z

After some testing I figured there should be an option to disable SMTP_LOGIN and SMTP_PASSWORD as it will block access to local smtp server without authentication.

Does this work?

services.mastodon.extraConfig = {
    SMTP_LOGIN = "";
    SMTP_PASSWORD = "";
};

If not, please share your local smtp server configuration and steps to reproduce the problem.

happy-river · 2020-08-13T18:38:49Z

But it's not possible to merge those definition, no? It would be fine we could merge all those definitions some somehow...

Let's continue the discussion at #83499.

Church- · 2020-09-15T18:27:43Z

@happy-river I'm curious if you've had trouble generating user accounts? Using your module I repeatedly get "This email is invalid" messages when trying to sign up.

Curious what you did?

I'm using a mail server on a vps outside my lan.
But I've also attempted to use Gmail's smtp server too to no change.

nyanloutre · 2020-11-09T11:37:41Z

I'm using this module for a long time now, what is blocking it from getting merged ?

mkg20001 · 2020-11-11T03:07:43Z

Currently there's a merge conflict with all-tests.nix, likely since the line got modified in another commit. Should be easy to resolve.

Besides that are there any current blockers for this PR?

Additionally to running ./update.sh --ver v3.3.0 --patches ./resolutions.patch I did - Resolve merge conflicts in resolutions.patch - Specified buildInputs for the ruby gem openssl - Use ruby_2_7, as it is recommended in the release notes [0] - build mastodon-assets with RAILS_ENV, as it seems to be required. It would probably be nice to set it to production someday, but atm it is set to development, as production requires OTP_SECRET to be set. [0] https://github.com/tootsuite/mastodon/releases/tag/v3.3.0

erictapen · 2020-12-30T19:06:11Z

I just upgraded to v3.3.0. I tested this on a small instance and it seems to work. This time the upgrade included some manual steps, see the commit message for details.

I hope this is ok for everyone, that I push to this PR? If anybody uses this in production, you probably need to follow the migration instructions in the release notes.

erictapen

This PR and it's predecessor has been waiting for a long time and has seen a fair bit of community involvement. As far as I followed the discussion the only current issue (besides a trivial merge conflict) is the increase in closure size, due to lack of duplication in yarn.nix.

As the metric for the problem is not the size of the Nixpkgs git repository but the size of the resulting tarball, I'd say this is a problem that could still be solved after this PR is merged.

So I'd vouch to get this merged. It is already getting used by people, it is getting maintained. Both of it would be easier if #78810 would be merged. I won't do it myself though, as I only recently aquired the commit bit and don't trust my judgement enough on this controversial PR.

leo60228 · 2021-01-11T15:35:55Z

I get this error when building:

Building native extensions. This could take a while...
ERROR:  Error installing /nix/store/2ysdwqd7a3ahp7dmwqxgizry3zpxfqx1-bcrypt-3.1.16.gem:
        ERROR: Failed to build gem native extension.

    current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
/nix/store/sxvw9ja9i1ndxslrbkird76m3jsamjsg-ruby-2.7.2/bin/ruby -I /nix/store/sxvw9ja9i1ndxslrbkird76m3jsamjsg-ruby-2.7.2/lib/ruby/2.7.0 -r ./siteconf20210111-8-3yq1qn.rb extconf.rb
creating Makefile

current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
make "DESTDIR=" clean
current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
make "DESTDIR="
make failedNo such file or directory - make "DESTDIR="

Gem files will remain installed in /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16 for inspection.
Results logged to /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/bcrypt-3.1.16/gem_make.out

Nothing I've tried is able to reproduce this other than just building the ruby-bcrypt derivation. breakpointHook works, but I can't find any more details on the issue anywhere in the filesystem.

leo60228 · 2021-01-11T16:20:08Z

I had a broken sandbox-paths in my nix.conf, not sure how I didn't catch that until now...

leo60228 · 2021-01-23T18:40:45Z

Make pkgs.mastodon.override { ... } work to build forks of Mastodon.

To make this a bit easier, perhaps update.sh should create an override.nix that's something like this?

{ mastodon, callPackage }:
mastodon.override {
  version = import ./version.nix;
  srcOverride = callPackage ./source.nix {};
  dependenciesDir = ./.;
}

bqv · 2021-02-02T17:07:54Z

Merge conflict :)

erictapen · 2021-02-11T21:11:58Z

As nobody else joined in my initiative to get this finally merged (either by approving or merging it), I deduce consensus is not strong enough among the commiters. Still there seems to be great interest in having this. So let's have it as an out-of-tree module?

I started working on a Nix Flake mastodon-nixos, that currently somewhat mirrors the functionality of this PR. I intend to further maintain this Flake and add more functionality in the future (e.g. support for non-flake users). Goal is to move the project into the nix-community org, as I guess that'd be the right place for it. Could a member add me to the org so I can push the repo?

Also I will close this PR now. It is currently based on an commit from August 2020. That means that it uses an old version of ffmpeg and imagemagick, whose should have accumulated multiple security vulnerabilities by now. No one should use this branch without a rebase in production. Feel free to reopen if anyone wants to maintain it, but I currently see not much future in it.

Edit: Got it accepted into nix-community! https://github.com/nix-community/mastodon-nixos

Edit: as #112898 just got merged, there is no need for an out-of-tree module anymore. I'll delete the repository.

bqv · 2021-02-12T01:11:36Z

Oh gosh, thank you @erictapen ! A flake is a great idea

Izorkin · 2021-02-12T07:58:06Z

How to fix this error?:

mastodon-init-dirs-start[91340]: ERROR: Missing RAILS_ENV environment variable, please set it to "production", "development", or "test".

and

mastodon-init-db.service: Failed to load environment files: No such file or directory

erictapen · 2021-02-12T12:38:24Z

So as it turns out there are some people from the nix-community IRC willing to merge this.
I rebased on current master. For that I had to open a new PR at #112898.

happy-river requested a review from infinisil as a code owner January 30, 2020 05:17

happy-river changed the title ~~Feature/mastodon~~ Add Mastodon package and module Jan 30, 2020

ofborg bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Jan 30, 2020

happy-river mentioned this pull request Jan 30, 2020

add mastodon package and module #60788

Closed

20 tasks

ofborg bot added the 8.has: package (new) This PR adds a new package label Jan 30, 2020

ofborg bot requested a review from a user January 30, 2020 05:35

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Jan 30, 2020

ghost reviewed Jan 30, 2020

View reviewed changes

nixos/modules/services/web-apps/mastodon.nix Outdated Show resolved Hide resolved

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Feb 2, 2020

ghost mentioned this pull request Feb 4, 2020

language package management tooling missing hashes #65275

Open

happy-river force-pushed the feature/mastodon branch from b094bd6 to 3c4dad5 Compare February 8, 2020 15:41

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Feb 8, 2020

midchildan mentioned this pull request Sep 18, 2020

rust-analyzer: 2020-08-24 -> 2020-09-14 and some clean-up #97444

Merged

10 tasks

ryantm added 2.status: merge conflict This PR has merge conflicts with the target branch and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels Oct 3, 2020

erictapen added 2 commits December 20, 2020 03:11

mastodon: 3.2.0 -> 3.2.2

f2d7b94

happy-river requested a review from alyssais as a code owner December 30, 2020 19:02

ofborg bot added the 6.topic: ruby label Dec 30, 2020

erictapen approved these changes Jan 1, 2021

View reviewed changes

erictapen closed this Feb 11, 2021

erictapen mentioned this pull request Feb 12, 2021

mastodon,nixos/mastodon: init at 3.3.0 #112898

Merged

10 tasks

erictapen added a commit to erictapen/nixpkgs that referenced this pull request Dec 22, 2021

test from NixOS#78810

27a9375

vivlim added a commit to vivlim/nixpkgs that referenced this pull request Nov 6, 2022

Copy in documentation written by happy-river in NixOS#78810

6705eb0

vivlim added a commit to vivlim/nixpkgs that referenced this pull request Nov 6, 2022

Copy in documentation written by happy-river in NixOS#78810

59a6810

Janik-Haag added the 12. first-time contribution This PR is the author's first one; please be gentle! label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Mastodon package and module #78810

Add Mastodon package and module #78810

happy-river commented Jan 30, 2020

ghost left a comment •

edited by ghost

Loading

Mic92 commented Jan 30, 2020

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 •

edited by ghost

Loading

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 •

edited by ghost

Loading

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 •

edited by ghost

Loading

Mic92 commented Jan 30, 2020 •

edited

Loading

happy-river commented Jan 31, 2020

ghost commented Jan 31, 2020

happy-river commented Jan 31, 2020

nyanloutre commented Feb 1, 2020 •

edited

Loading

happy-river commented Feb 8, 2020

happy-river commented Aug 13, 2020

Church- commented Sep 15, 2020 •

edited

Loading

nyanloutre commented Nov 9, 2020

mkg20001 commented Nov 11, 2020

erictapen commented Dec 30, 2020

erictapen left a comment

leo60228 commented Jan 11, 2021

leo60228 commented Jan 11, 2021

leo60228 commented Jan 23, 2021

bqv commented Feb 2, 2021

erictapen commented Feb 11, 2021 •

edited

Loading

bqv commented Feb 12, 2021

Izorkin commented Feb 12, 2021

erictapen commented Feb 12, 2021

Add Mastodon package and module #78810

Add Mastodon package and module #78810

Conversation

happy-river commented Jan 30, 2020

Motivation for this change

Things done

ghost left a comment • edited by ghost Loading

Choose a reason for hiding this comment

Mic92 commented Jan 30, 2020

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 • edited by ghost Loading

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 • edited by ghost Loading

Mic92 commented Jan 30, 2020

ghost commented Jan 30, 2020 • edited by ghost Loading

Mic92 commented Jan 30, 2020 • edited Loading

happy-river commented Jan 31, 2020

ghost commented Jan 31, 2020

happy-river commented Jan 31, 2020

nyanloutre commented Feb 1, 2020 • edited Loading

happy-river commented Feb 8, 2020

happy-river commented Aug 13, 2020

Church- commented Sep 15, 2020 • edited Loading

nyanloutre commented Nov 9, 2020

mkg20001 commented Nov 11, 2020

erictapen commented Dec 30, 2020

erictapen left a comment

Choose a reason for hiding this comment

leo60228 commented Jan 11, 2021

leo60228 commented Jan 11, 2021

leo60228 commented Jan 23, 2021

bqv commented Feb 2, 2021

erictapen commented Feb 11, 2021 • edited Loading

bqv commented Feb 12, 2021

Izorkin commented Feb 12, 2021

erictapen commented Feb 12, 2021

ghost left a comment •

edited by ghost

Loading

ghost commented Jan 30, 2020 •

edited by ghost

Loading

ghost commented Jan 30, 2020 •

edited by ghost

Loading

ghost commented Jan 30, 2020 •

edited by ghost

Loading

Mic92 commented Jan 30, 2020 •

edited

Loading

nyanloutre commented Feb 1, 2020 •

edited

Loading

Church- commented Sep 15, 2020 •

edited

Loading

erictapen commented Feb 11, 2021 •

edited

Loading