Commit Graph

506 Commits (8e84ebf0cb211c1d94145399b05c9f2ad0e4d4b0)

Author SHA1 Message Date
Claire 82caed594c
Change deduplication order of tootctl maintenance fix-duplicates (#15923)
Hopefully fixes #15922

Also update support up to latest database schema version
4 years ago
Claire b358229834
Further preparation for Rails 6 (#15916)
* Use ActiveRecord::Result#to_ary instead of deprecated to_hash

They do the same thing, and to_hash has been removed from Rails 6.1

* Explicitly name polymorphic indexes to workaround a bug in Rails 6.1

cf. https://github.com/rails/rails/issues/41693

* Fix incorrect usage of “foreign_key” in migration script

* Use `ActiveModel::Errors#delete` instead of deprecated clear method

* Fix link headers tests on Rails 6.1

Rails 6.1 adds values to the Link header by default, thus it is not a
LinkHeader object anymore. Fix the test to parse the Link header instead
of assuming it is a LinkHeader.
4 years ago
Claire 9aaaa96d2f
Use more robust hook for loading timestamp_id function into database (#15919) 4 years ago
Claire a4dcaef53b
Prepare Mastodon for zeitwerk autoloader (#15917)
* Prepare Mastodon for zeitwerk autoloader (Rails 6)

Add inflections and rename/move a few classes.

In particular, app/lib/exceptions.rb and app/lib/sanitize_config.rb
were manually loaded while still in autoload paths.

* Add inflection for Url → URL
4 years ago
Claire 43eff898a0
Prepare Mastodon for Rails 6 (#15911)
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
4 years ago
Claire a8139ab016
Add borders to 📱 and 📲 emojis (#15794) 4 years ago
Justin Tracey c9e8e1739c
replace all instances of "ends_with?" with "end_with?" (#15745)
The "ends_with?" method is just a Rails alias of Ruby's "end_with?" method.
Using the latter makes the code less brittle.
4 years ago
Justin Tracey 3f8523130d
use host instead of headers to make Rack happy (#15741)
"headers" is provided by Rails, Rack can't rely on it
4 years ago
Cecylia Bocovich 3447bd2f80
Monkey patch Rack::Session to send secure cookies to onions (#15725) 4 years ago
Claire 21fb3f3684
Drop dependency on secure_headers, fix response headers (#15712)
* Drop dependency on secure_headers, use always_write_cookie instead

* Fix cookies in Tor Hidden Services by moving configuration to application.rb

* Instead of setting always_write_cookie at boot, monkey-patch ActionDispatch
4 years ago
Claire 6feaad33f2
Fix obsolete digitalocean.rake file breaking rake tasks (#15618) 4 years ago
Shubhendra Singh Chauhan c8d11b8bdb
Fixed code quality issues (#15541)
* Added .deepsource.toml

* Removed bad use of `alias`

* Fixed operand order in the binary expression

* Prefixed unused method arguments with an underscore

* Replaced the old OpenSSL algorithmic constants with the newer strings initializers.

* Removed unnecessary UTF-8 encoding comment
4 years ago
ThibG 13d5b81579
Fix “tootctl accounts unfollow” (#15639)
Fixes #15635

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
luigi eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
4 years ago
ThibG 54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
4 years ago
ThibG 9395143126
Fix maintenance script not re-indexing some indexes on textual values (#15515)
* Fix maintenance script not re-indexing some indexes on textual values

Fixes #15475

* Refresh instance view at the end of the maintenance script run

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
abcang efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
4 years ago
Sophie Parker c7262d12b2
Improve Emoji import (fix #15429) (#15430)
* Improve Emoji import

Skip macOS '._' shadow files in tar archive to speed up import

* Fix codeclimate format issue with whitespace

* Update lib/mastodon/emoji_cli.rb

suggestions from Gargron to improve comment

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>

* Update emoji_cli.rb

Remove extraneous comment (macOS-specific comment now with correct line)

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
4 years ago
Eugen Rochko 633d175146
Bump version to 3.3.0 (#15433) 4 years ago
ThibG f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
Eugen Rochko 444b21b55f
Bump version to 3.3.0rc3 (#15411) 4 years ago
ThibG 1cf2c3a810
Fix external user creation failing when invite request text is required (#15405)
* Fix external user creation failing when invite request text is required

Also fixes tootctl-based user creation.

* Add test about invites when invite request text is otherwise required

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
Eugen Rochko 9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 4 years ago
ThibG c4e860277d
Fix "tootctl accounts fix-duplicates" (#15373)
- `pluck_each` cannot be used this way with `group`
- typo

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
Eugen Rochko c6598b17d9
Bump version to 3.3.0rc2 (#15365) 4 years ago
ThibG a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
ThibG 75944a2f2d
Bump max supported schema version in maintenance script (#15359)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
ThibG 8485c436d5
Bump max supported schema version in maintenance script (#15349)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
4 years ago
Eugen Rochko 1045549f85
Add stoplight for object storage failures, return HTTP 503 (#13043) 4 years ago
Eugen Rochko 216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
4 years ago
ThibG fbe7519243
Fix account merging in maintenance script (#15264)
Also include AccountNote and AccountDeletionRequest to the list of classes
needing to be reassigned the merged account.
4 years ago
ThibG fbff89be03
Fix tootctl ip_blocks add failing if not given a comment (#15263)
Fixes #15261
4 years ago
Eugen Rochko 8e0b98556f
Bump version to 3.3.0rc1 (#15230) 4 years ago
Eugen Rochko e7f1439b97
Bump version to 3.2.1 (#15227) 4 years ago
Eugen Rochko ec0edf74d6
Fix post-migration warning about index corruption being too scary (#15223) 4 years ago
Eugen Rochko f844386809
Add `tootctl accounts merge` (#15201)
* Add `tootctl accounts merge`

* Update lib/mastodon/accounts_cli.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
4 years ago
ThibG 1242e57c27
Deal with collation-related index corruption (#14860)
* Add tootctl maintenance fix-duplicates

This tool goes through the database to detect and fix duplicates.
This operation is very slow and may cause data loss (of data that would be
inaccessible without intervention because of the existing index corruptions).
It tries its best to make sensible decisions, and asks the user in some cases.

* Add warning message in db:migrate hook

* Clear Rails cache after being done with database deduplication

Avoids followers hash cache being incorrect, among other things
4 years ago
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 4 years ago
Eugen Rochko 9b1f2a4b61
Add subresource integrity for JS and CSS assets (#15096)
Fix #2744
4 years ago
Eugen Rochko 5e1364c448
Add IP-based rules (#14963) 4 years ago
ThibG 5bbc9a4f78
Fix downloading remote media files when server returns empty filename (#14867)
Fixes #14817
4 years ago
Eugen Rochko ed099d8bdc
Change account suspensions to be reversible by default (#14726) 4 years ago
Eugen Rochko e6b272e5c9
Change REST API to return empty data for suspended accounts (#14765) 4 years ago
ThibG 91eecd1b3c
Add border around 🕺 emoji (#14769)
Fixes #14768
4 years ago
Y.Yamashiro abc91e5fa6
Change S3 ACL in "tootctl media remove-orphans --fix-permissions" from fixed value to environment file's value. (#14715) 4 years ago
Eugen Rochko 71085987e8
Add `--fix-permissions` option to `tootctl media remove-orphans` (#14383) 4 years ago
dependabot[bot] 8972e5f7f6
Bump rubocop from 0.86.0 to 0.88.0 (#14412)
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
4 years ago
ThibG f0b6ddd979
Add border around 🐞 emoji (#14712) 4 years ago
Takeshi Umeda 175cd4f8ef
Fix an error when file_file_size is nil in tootctl media remove (#14657) 4 years ago
ThibG 8d217d7231
Improve email address validation (#14565)
* Increase DNS timeout from 1 second to 5 seconds for MX check

1 seconds is rather short when using a recursive DNS resolver which
hasn't got a cached result already available. Use 5 seconds instead,
which is the timeout value we use for outgoing HTTP queries.

* Add more precise error messages for invalid e-mail addresses
5 years ago
ThibG decc5b9a78
Fix `tootctl media` commands not handling snowflake ids for media_attachments (#14536) 5 years ago
ThibG 0ebc8d8626
Fix thumbnail color extraction (#14464)
* Fix contrast calculation for thumbnail color extraction

Luminance calculation was using 0-255 RGB values instead of 0-1 sRGB values,
leading to incorrectly-computed contrast values.

Since we use ColorDiff already, just use its XYZ colorspace conversion code
to get the value.

* Require at least 3:1 contrast for both accent and foreground colors

* Lower required contrast for the accent color
5 years ago
ThibG a1412491b7
Change content-type to be always computed from file data (#14452)
* Change content-type to be always computed from file data

Restore previous behavior, detecting the content-type isn't very
expensive, and some instances may serve files as application/octet-stream
regardless of their true type, making fetching media from them fail, while
it used to work pre-3.2.0.

* Add test
5 years ago
Eugen Rochko 93dd413a47
Bump version to 3.2.0 (#14395) 5 years ago
Eugen Rochko d1a6f8d5a8
Bump version to 3.2.0rc2 (#14382) 5 years ago
Takeshi Umeda 101485a41f
Fix mimetype returning nil (#14356) 5 years ago
Eugen Rochko 6f798eb574
Bump version to 3.2.0rc1 (#14312) 5 years ago
Eugen Rochko 4abe3be321
Change `tootctl search deploy` algorithm (#14300) 5 years ago
Eugen Rochko 96e89d1ef4
Bump version to 3.1.5 (#14284) 5 years ago
ThibG 701e5b9a19
Fix ogg vorbis files with a cover art not being correctly processed (#14255) 5 years ago
Eugen Rochko 99f3a55540
Add color extraction for audio thumbnails (#14209) 5 years ago
Eugen Rochko 534da9ba23
Fix audio uploads without embedded image (#14203) 5 years ago
Eugen Rochko 6d23d40420
Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 5 years ago
Eugen Rochko 7aaf2b44ec
Fix remote files not using Content-Type header, streaming (#14184) 5 years ago
Eugen Rochko 64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
5 years ago
Eugen Rochko 8c04e37b03
Remove the terms blacklist and whitelist from UX (#14149)
Localization strings:

- "Whitelist mode" -> "Limited federation mode"
- "Blacklist e-mail domain" -> "Block e-mail domain"
- "Whitelist domain" -> "Allow domain for federation"

...And so on

Environment variables (backwards-compatible):

- `WHITELIST_MODE` -> `LIMITED_FEDERATION_MODE`
- `EMAIL_DOMAIN_BLACKLIST` -> `EMAIL_DOMAIN_DENYLIST`
- `EMAIL_DOMAIN_WHITELIST` -> `EMAIL_DOMAIN_ALLOWLIST`

tootctl:

- `tootctl domains purge --whitelist-mode` -> `tootctl domains purge --limited-federation-mode`

Removed badly maintained and no longer relevant .env.production.sample file
5 years ago
Eugen Rochko 4662afe075
Fix help text around `tootctl email_domain_blocks` (#14147) 5 years ago
tateisu 370e38ee15
Add tootctl email-domain-blocks (#13589)
* Add tootctl email_domains (block|unblock)

* fix codeclimate issues.

* fix codeclimate issues.

* fix codeclimate issues.

* add list subcommand, remove log_action.

* fix codeclimate issues.

* filter duplicate hostnames,ips before block

* rebase from currnet master branch.
rename email_domains_cli.rb to email_domain_blocks_cli.rb .
rename Mastodon::EmailDomainsCLI to Mastodon::EmailDomainBlocksCLI .
rename command email_domains to email-domain-blocks . (Thor recognizes both of - and _ )
rename subcommand block to add .
rename subcommand unblock to remove .
change the color in list subcommand to while for domain or cyan for childlen.
don't use include() in list subcommand.
suppress console output about succeeded entry.
add console output about count of processed/skipped.
remove capitalization in subcommand description.
remove long_desc in subcommand 'remove'.
remove duplicate where in subcommand 'remove'.

* fix codeclimate issue.
5 years ago
leo60228 e0f55f374c
Add emojis:generate_borders Rake task (#13773)
* Add emojis:generate_borders Rake task

* Address review

* Border all dark emoji

* Combine stroke with filter to reduce artifacting

* Cleanup Camera with Flash

* Add stroke-linejoin="round"

The previous filter and tweaks were effectively a poor imitation of it.
There are no artifacts for any dark emoji now!

* Set stroke-width using property

This fixes old versions of Firefox.

* Store emoji in string instead of array

* Use separate arguments for each path segment

* Remove "background: black;"
5 years ago
ThibG 346d9b3d2e
Add blurhash to preview cards (#13984)
Fixes #13001
5 years ago
ThibG 175fd5b8ba
Fix mastodon:setup choking on env variables containing '%' (#13940)
Fix a regression introduced in #13928, caused by TTY::Command building
shell commands by chaining string substitutions.

Ditch TTY::Command and use system instead (both do shell out).
5 years ago
ThibG 9b7e3b4774
Fix mastodon:setup not running properly when some environment variables are set (#13928)
When using one of the docker-compose containers, mastodon:setup will use the
existing .env.production rather than the generated one during the setup steps.

This is because dotenv does not overwrite env variables that are alreayd
defined, and the docker-compose.yml file loads the environment variables
from .env.production.
5 years ago
Stanislas ce87469d0a
Fix `tootctl upgrade storage-schema` S3 ACL (#13768) 5 years ago
Eugen Rochko 199bbbcb9f
Fix `tootctl media remove-orphans` choking on unknown files in storage (#13765)
Fix #13762

Catch tootctl interrupt to prevent confusing stacktrace
5 years ago
Eugen Rochko 2b91a3dac0
Fix `tootctl upgrade storage-schema` misbehaving (#13761)
- Fix not moving original files of custom emojis
- Fix command failing to move any files with S3 storage
- Fix command marking records as upgraded when move failed

Fix #13594
5 years ago
ThibG 927f9ea499
Fix GifReader exceptions (#13760) 5 years ago
Eugen Rochko 06225dee94
Bump version to 3.1.4 (#13750) 5 years ago
Takeshi Umeda cbfb2c0e71
Fix tootctl media refresh (#13751) 5 years ago
ThibG 34756cc4e0
Fix "tootctl media remove-orphans" crashing on “Import” files (#13685)
* Fix "tootctl media remove-orphans" crashing on “Import” files

* Also remove empty directories when removing orphaned media
5 years ago
Hanage999 ff72c0472f
Fix tootctl upgrade storage-schema failing to delete empty directories (#13593) 5 years ago
Lerk 157850577f
Add `tootctl emoji export` (#13534)
* add emoji export command to cli

* fix codeclimate issues

* add error when no matching category was found

* add other suggestions

* exit 1 when no matching category is found

* changes according to suggestions

* 👀

* RubyNein

Y u always autoformat :c
5 years ago
Eugen Rochko c3ca3801f2
Add separate cache directory for non-local uploads (#12821) 5 years ago
D Anzorge 29f5353f8b
Update Twemoji to 12.1.5 (#13021)
* Update Twemoji to 12.1.5

SVGs from Twemoji repo https://github.com/twitter/twemoji

* Update emoji_map.json to Unicode 12
5 years ago
Eugen Rochko c99bc5d51b
Fix regression in `tootctl media remove-orphans` (#13405)
Fix #13401
5 years ago
Eugen Rochko 0773a512ac
Fix trying to delete already deleted file when post-processing (#13406)
Fix #13403
5 years ago
Eugen Rochko a889756dd5
Bump version to 3.1.3 (#13389) 5 years ago
Eugen Rochko a9a063c0e9
Fix `tootctl media remove-orphans` ignoring `PAPERCLIP_ROOT_PATH` (#13375)
Fix #13371
5 years ago
ThibG 6c79b7237e
Fix Paperclip using deprecated URI.escape function (#13320)
Monkey-patch Paperclip to perform URL escaping in a slightly more
appropriate way, and get rid of runtime deprecation warnings.
5 years ago
Eugen Rochko 0c8945e5ff
Change `tootctl media remove-orphans` to work for all classes (#13316)
Change `tootctl media lookup` to not use an interactive prompt
5 years ago
ThibG 2c6099125d
Fix videos with unsupported colorspace not being transcoded (#13242) 5 years ago
ThibG abd8394880
Fix MP4 (H264 + AAC) video files being needlessly re-encoded (#13239) 5 years ago
Eugen Rochko 9660aa4543
Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
5 years ago
tateisu 4e524218d3
Add `--skip-media-remove` option to `tootctl statuses remove` (#13080)
* Add skip_media_remove option to tootctl statuses remove

* Add skip_media_remove option to tootctl statuses remove

Co-authored-by: tateisu <tateisu@juggler.jp>
5 years ago
ThibG ce17cea221
Fix installation failing when Redis password contains special characters (#13156)
* Add support for special characters in Redis passwords

Fixes #13154

* Refactor
5 years ago
Eugen Rochko aa8568b7f2
Bump version to 3.1.2 (#13162) 5 years ago
ThibG ca7ee9f1cf
Add `--reset-password` option to `tootctl accounts modify` (#13126) 5 years ago
Marcin Cieślak bd4bbba98a
Do not suggest to login into the GitLab database (#13084)
As pointed out on Discourse:

https://discourse.joinmastodon.org/t/obscure-wtf-error-message-running-migrations-for-3-1/2524

The message the administrators were getting was telling them
to log in to the "GitLab database" and also mentions MySQL
5 years ago
Eugen Rochko b752666e3f
Bump version to 3.1.1 (#13069) 5 years ago
Eugen Rochko d2bcef7f6f Bump version to 3.1.0 (#13053) 5 years ago