Commit Graph

654 Commits (42e733681aef45c9e6224aed288e4c845782c173)

Author SHA1 Message Date
ThibG 7f5a4be580 Fix emoji update date processing (#9255) 6 years ago
ThibG 9b89c62d43 Check that twitter:player is valid before using it (#9254)
Fixes #9251
6 years ago
Eugen Rochko 6d59dfa15d
Optimize the process of following someone (#9220)
* Eliminate extra accounts select query from FollowService

* Optimistically update follow state in web UI and hide loading bar

Fix #6205

* Asynchronize NotifyService in FollowService

And fix failing test

* Skip Webfinger resolve routine when called from FollowService if possible

If an account is ActivityPub, then webfinger re-resolving is not necessary
when called from FollowService. Improve options of ResolveAccountService
6 years ago
m.b cd41c2c6ad Update resolve_url_service.rb (#9188) 6 years ago
James Kiesel 4c03e05a4e Allow joining several hashtags in a single column (#8904)
* Nascent tag menu on frontend

* Hook up frontend to search

* Tag intersection backend first pass

* Update yarnlock

* WIP

* Fix for tags not searching correctly

* Make radio buttons function

* Simplify radio buttons with modeOption

* Better naming

* Rearrange options

* Add all/any/none functionality on backend

* Small PR cleanup

* Move to service from scope

* Small cleanup, add proper service tests

* Don't use send with user input :D

* Set appropriate column header

* Handle auto updating timeline

* Fix up toggle function

* Use tag value correctly

* A bit more correct to use 'self' rather than 'all' in status scope

* Fix some style issues

* Fix more code style issues

* Style select dropdown more better

* Only use to_id'ed value to ensure no SQL injection

* Revamp frontend to allow for multiple selects

* Update backend / col header to account for more flexible tagging

* Update brakeman ignore

* Codeclimate suggestions

* Fix presenter tag_url

* Implement initial PR feedback

* Handle additional tag streaming

* CodeClimate tweak
6 years ago
Eugen Rochko ce2ee68b64
Revert "Fix FetchAtomService content type handling (#9132)" (#9171)
This reverts commit c36a4a1617.
6 years ago
valerauko c36a4a1617 Fix FetchAtomService content type handling (#9132)
* Add profile to json+ld in Accept

It's required by the ActivityPub spec

* Use headers['Content-type'] instead of mime_type

mime_type strips the profile from the content type, but it's still available raw in the headers hash

* Add test for ld+json with profile
6 years ago
Eugen Rochko be202f9377
Accept the same payload in multiple inboxes and deliver (#9150) 6 years ago
Eugen Rochko 47b8d195e6
Always let through notifications from staff (#9152)
* Always let through notifications from staff

Follow-up to #8993

* Let messages from staff through, but no other notifications
6 years ago
ThibG a03d506626 Fix Pleroma mentions being fetched as preview cards (#9158) 6 years ago
ThibG 33a71e8f7c Do not hide boost notifications from followed people with hidden boosts (#9147)
* Do not hide boost notifications from followed people with hidden boosts

Not displaying boosts from a followed user in the Home timeline and not
having notifications when they reblog your own content are two very
separate concerns, tying them together seem counter-intuitive and unwanted.

* Update specs accordingly
6 years ago
Eugen Rochko 795f0107d2
Include preview cards in status entity in REST API (#9120)
* Include preview cards in status entity in REST API

* Display preview card in-stream

* Improve in-stream display of preview cards
6 years ago
Eugen Rochko eef8d9a5f7
Add locality check to ActivityPub::FetchRemoteAccountService (#9109)
* Add locality check to ActivityPub::FetchRemoteAccountService

Fix #8643

Because there are a few places where it is called, it is difficult
to confirm if they all previously checked it for locality. It's better
to make sure within the service.

* Remove faux-remote duplicates of local accounts
6 years ago
abcang 1e2695198a Skip link-back check if body is nil (#9107) 6 years ago
ThibG 7fee968e9f Do not fetch preview card for mentioned users (#6934) 6 years ago
Eugen Rochko ddd30f331c
Improve support for aspects/circles (#8950)
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
6 years ago
Quint Guvernator f5e2e96e95 always allow DMs from staff (#8993) 6 years ago
Eugen Rochko 87fdd139b8
Do not push DMs into the home feed (#8940)
* Do not push DMs into the home feed

* Show DMs column after sending a DM, if DMs column is not already shown
6 years ago
Eugen Rochko 774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
6 years ago
Eugen Rochko 7fe137d2f7
Fix link verification for remote accounts (#8868) 6 years ago
Eugen Rochko f92f1ee80a
Support link verification with redirects (#8735)
(e.g. URL shortener)
6 years ago
Yamagishi Kazutoshi 3da1cc7d5e Fix failed profile verification when rel attribute including values other than me (#8733) 6 years ago
Eugen Rochko 68833a50d4
Fix VerifyAccountLinksWorker not being queued (#8721)
UX-wise, people expect that saving the profile will re-check links even without changing fields content. Bug-wise, `@account` was undefined.

Regression from #8703
6 years ago
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
6 years ago
ThibG bd9e47e9be Handle relative URLs when fetching OEmbed/OpenGraph cards (#8669) 6 years ago
Eugen Rochko c593d6df9c
Add preference for report notification e-mails, skip for duplicates (#8559)
If an unresolved report for the same target account already exists,
no new notification is generated
6 years ago
Eugen Rochko cabdbb7f9c
Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
7 years ago
Eugen Rochko 8adf485c0f
Bump interactions count when a follow happens (#8468) 7 years ago
Eugen Rochko 55c856c522
Delay e-mail notifications by 2 minutes (#8438)
Fix #8430
7 years ago
ThibG 42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes #8376) (#8377) 7 years ago
Eugen Rochko 802cf6a4c5
Improve federated ID validation (#8372)
* Fix URI not being sufficiently validated with prefetched JSON

* Add additional id validation to OStatus documents, when possible
7 years ago
Eugen Rochko 1b282299df
Add favourites to archive takeout (#8351)
Remove experimental key export
7 years ago
Eugen Rochko d98de8ada7
Get rid of all batch order warnings (#8334) 7 years ago
Eugen Rochko 6226aa83d7
Increase reach of Delete->Actor activities (#8305)
Fix #7316
7 years ago
ThibG abc5548cca Do not process outgoing mentions to suspended accounts (#8272) 7 years ago
ThibG 1ee675d68b Use correct activity id in Accept when receiving duplicate Follow (fixes #8218) (#8244) 7 years ago
ThibG af912fb308 Allow accessing local private/DM messages by URL (#8196)
* Allow accessing local private/DM messages by URL

(Provided the user pasting the URL is authorized to see the toot, obviously)

* Fix SearchServiceSpec tests
7 years ago
Eugen Rochko cc94b1d95a Prevent ActivityPub movedTo recursion (#8092)
Fix #8051
7 years ago
Eugen Rochko a23ac107e4
Fix activity:interactions counter to count all interactions (#8037) 7 years ago
Eugen Rochko e55dce3176
Add federation relay support (#7998)
* Add federation relay support

* Add admin UI for managing relays

* Include actor on relay-related activities

* Fix i18n
7 years ago
Eugen Rochko 8461cd4bb5
Send undo of boost to original poster if reblog (#7959)
Fix #7874
7 years ago
Eugen Rochko b0968623fa
Fix check for PotentialFriendshipTracker when replying (#7933) 7 years ago
Eugen Rochko da8fe8079e
Re-add follow recommendations API (#7918)
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
7 years ago
Eugen Rochko a58ec29631
Allow selecting default posting language instead of auto-detect (#7828)
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
7 years ago
Eugen Rochko ca85658975
Add autofollow option to invites (#7805)
* Add autofollow option to invites

* Trigger CodeClimate rebuild
7 years ago
Eugen Rochko c3b3594305
Remove placeholder text for media-only toots (#7806)
Ref: cfa9b6e13a

This breaks compatibility with pre-2.3.0 Mastodon and older
software, but at the time of writing the network is >80% above
that version.

Compatibility broken only for toots with no text.
7 years ago
Eugen Rochko 10f51c9886
Fix domain hiding logic (#7765)
* Send rejections to followers when user hides domain they're on

* Use account domain blocks for "authorized followers" action

Replace soft-blocking (block & unblock) behaviour with follow rejection

* Split sync and async work of account domain blocking

Do not create domain block when removing followers by domain, that
is probably unexpected from the user's perspective.

* Adjust confirmation message for domain block

* yarn manage:translations
7 years ago
Yamagishi Kazutoshi e3fb528d12 Skip processing when HEAD method returns 501 (#7730) 7 years ago
Akihiko Odaki 5bf5003384 Do not mark remote status sensitive even if spoiler text is present (#7395)
Old statuses and statuses from Pawoo, which runs a modified version of
Mastodon, may not have been marked sensitive even if spoiler text is
present.

Such statuses are still not marked sensitve if they are local or
arrived before version upgrade. Marking recently fetched remote status
sensitive contradicts the behavior.

Considering what people expected when they authored such statuses, this
change removes the sensitivity enforcement.
7 years ago
Eugen Rochko 1e938b966e
Exclude unlisted, private and direct toots from affecting trends (#7686) 7 years ago
Eugen Rochko a7d726c383
Improve counter caches on Status and Account (#7644)
Do not touch statuses_count on accounts table when mass-destroying
statuses to reduce load when removing accounts, same for
reblogs_count and favourites_count

Do not count statuses with direct visibility in statuses_count

Fix #828
7 years ago
Eugen Rochko 461542784b
Reduce wasted work in RemoveStatusService due to inactive followers (#7672) 7 years ago
Eugen Rochko 9bd23dc4e5
Track trending tags (#7638)
* Track trending tags

- Half-life of 1 day
- Historical usage in daily buckets (last 7 days stored)
- GET /api/v1/trends

Fix #271

* Add trends to web UI

* Don't render compose form on search route, adjust search results header

* Disqualify tag from trends if it's in disallowed hashtags setting

* Count distinct accounts using tag, ignore silenced accounts
7 years ago
Eugen Rochko 32d4372381
Use #any? instead of #exists? when checking media attachments (#7570)
If media_attachments are not loaded, SQL query is the same, but
the #exists? method performs SQL query even if preloaded
7 years ago
Yamagishi Kazutoshi 7403e5d306 Add media timeline (#6631) 7 years ago
Akihiko Odaki 55fd55714a Raise Mastodon::RaceConditionError if Redis lock failed (#7511)
An explicit error allows user agents to know the error and Sidekiq to
retry.
7 years ago
unarist d47091eb97 Fix custom emoji handling in UpdateRemoteProfileService (OStatus) (#7501)
This patch fixes NoMethodError and others in RemoteProfileUpdateWorker.
7 years ago
Wiktor 7fe2993b87 Fix account URI when updating ActivityPub account (#7488)
Updates account `uri` field on each call to `update_account` instead of
only once during `create_account` to mirror the same behavior in OStatus
`ResolveAccountService` class [0].

ActivityPub accounts are identified using `@username` and `@domain` pair
instead of URI since #6842.

This fixes #7479: a bug when the account identified by `@username` and
`@domain` changes its URI.

[0]:
03b69ebc45/app/services/resolve_account_service.rb (L121)
7 years ago
abcang 3793e598d0 Call media.present? because media may be nil (#7474) 7 years ago
ThibG 7467361d70 Fetch boosted statuses on behalf of a follower (fixes #7426) (#7459)
When an ActivityPub Announce is processed and the boosted toot is not known,
fetch it on behalf of one of the booster's followers. This is to allow
fetching self-boosts of previously-unknown private toots.

If fetching on behalf of a user fails, try fetching it anonymously: the
selected follower of a boosting user may be banned by the boosted toot's
author.
7 years ago
Eugen Rochko b4fb766b23
Add REST API for Web Push Notifications subscriptions (#7445)
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
7 years ago
Yamagishi Kazutoshi d2ee48977c Rescue Mastodon::LengthValidationError in FetchLinkCardService (#7424) 7 years ago
Surinna Curtis 01dfd6dbc8 Take the first recognized actor_type. (#7410) 7 years ago
ThibG 993e68a7dd Fix hashtags not being federated on mentions (fixes #6900) (#7406) 7 years ago
Eugen Rochko 42cd363542
Bot nameplates (#7391)
* Store actor type in database

* Add bot nameplate to web UI, add setting to preferences, API, AP
Fix #7365

* Fix code style issues
7 years ago
Eugen Rochko 0f0cc3f2eb
Support explicitly supplying language code for status via REST API (#7389) 7 years ago
Eugen Rochko c947e2e4c5
Fix handling of malformed ActivityPub payloads when URIs are nil (#7370)
* Fix handling of malformed ActivityPub payloads when URIs are nil

* Gracefully handle JSON-LD canonicalization failures
7 years ago
Eugen Rochko c73ce7b695
Store home feeds for 7 days instead of 14 (#7354)
* Store home feeds for 7 days instead of 14

Reduces workload for status fan-out to active followers

* Fix test for user model
7 years ago
Eugen Rochko 658cbc9425
Improve PostStatusService performance (#7317)
Offload creation of local notifications to a worker. Remove two
redundant SQL queries from ProcessMentionsService, remove n+1
XML/JSON serialization via memoization
7 years ago
Eugen Rochko cb5b5cb5f7
Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
7 years ago
abcang 71a7cea73f Keep notification when muting_notifications is true (#7311)
* Keep notification when muting_notifications is true

* Retrun mute object

* Fix test
7 years ago
Eugen Rochko f62539ce5c
Remove most behaviour disparities between blocks and mutes (#7231)
* Remove most behaviour disparities between blocks and mutes

The only differences between block and mute should be:

- Mutes can optionally NOT affect notifications
- Mutes should not be visible to the muted

Fix #7230
Fix #5713

* Do not allow boosting someone you blocked

Fix #7248

* Do not allow favouriting someone you blocked

* Fix nil error in StatusPolicy
7 years ago
Eugen Rochko 965345316f
Guard against nil URLs in Request class (#7284)
Fix #7265
7 years ago
Surinna Curtis dc786c0cf4 Support Actors/Statuses with multiple types (#7305)
* Add equals_or_includes_any? helper in JsonLdHelper

* Support arrays in JSON-LD type fields for actors/tags/objects.

* Spec for resolving accounts with extension types

* Style tweaks for codeclimate
7 years ago
Emelia Smith 495303d9b8 Prevent suspended accounts from appearing in AccountSearchService (#7246) 7 years ago
Eugen Rochko 75c4ab9d12
Remove "nsfw" category for sensitive statuses in OStatus serializer (#7048)
Fix #7011
7 years ago
Eugen Rochko ff87d1bc3e
Rescue SSL errors when processing mentions, remove useless line (#7184) 7 years ago
Kaito Sinclaire 156b916caf Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
7 years ago
ThibG 1c379b7ef4 Remove extra spaces from search API queries and public account headers (fixes #7129) (#7152) 7 years ago
Eugen Rochko 78ed4ab75f
Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
7 years ago
Eugen Rochko 778562c223
Ensure SynchronizeFeaturedCollectionWorker is unique and clean up (#7043)
* Ensure SynchronizeFeaturedCollectionWorker is unique and clean up

Fix #7041

* Fix code style issue
7 years ago
Renato "Lond" Cerqueira 0c52654b52 When creating status, if no sensitive status is given, use default (#7057)
Clients using the API that do not provide the sensitive flag are always
posting with false sensitive option.
7 years ago
Nolan Lawson 6ff3b3e4db Fix nil account issue in ProcessAccountService (#7019) 7 years ago
Eugen Rochko f890d2a766
Support all ActivityPub actor types (#6997)
Fix #6973
7 years ago
David Underwood 123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar (#6124)
Federate custom emojis with accounts
7 years ago
ThibG e573bb0990 Fix compatibility with PeerTube (#6968)
* Support fetching objects of convertible types by URL (fixes #6924)

* Ignore invalid hashtags
7 years ago
Akihiko Odaki 40e5d2303b Validate HTTP response length while receiving (#6891)
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
7 years ago
Akihiko Odaki 54b273bf99 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
7 years ago
Eugen Rochko 9381a7d9d5
Use username/domain to match existing accounts in ActivityPub (#6842)
See also: #6837, #6667
7 years ago
nightpool 566ace2d64 Add entropy to download filenames (#6811)
pretty quick fix, and with the 1 week expiration i don't think we need to be too worried about the existing files

closes #6798
7 years ago
Eugen Rochko 510c9049c7
For now, put a "." into no-text statuses with media for backcompat (#6691) 7 years ago
Eugen Rochko 64db9ed5f6
After blocking domain with reject_media, invalidate cache (#6679)
Media attachments are part of the association cache of statuses,
since they are presumed to be immutable. Unless this cache is
cleared manually, the statuses will continue to look like they
have media embedded.
7 years ago
Eugen Rochko cfa9b6e13a
Remove text requirement when media attached from statuses (#6672) 7 years ago
Eugen Rochko 9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
7 years ago
Aboobacker MK 49092945ab Fix 500 while searching after deleting a post (#6604)
Fixes #6602
7 years ago
Eugen Rochko 41a01bec23
Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
7 years ago
Eugen Rochko 61ed133fea
Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
7 years ago
Eugen Rochko f7765acf9d
Fix #5173: Click card to embed external content (#6471) 7 years ago
Eugen Rochko 3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
7 years ago
abcang 7e5c433dfc Fix saving of oEmbed image (#6409) 7 years ago