aiden
/
mastodon
mirror of https://github.com/mastodon/mastodon
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,977 Commits
135 Branches
281 Tags
1.1 GiB
Commit Graph

11 Commits (a40167cf4d51743965a90f1f7496b5a1e9e25f18)

Author SHA1 Message Date
Renato "Lond" Cerqueira ad207456d6 Improve language filter () 
* Scrub text of html before detecting language.

* Detect language on statuses coming from activitypub.

* Fix rubocop comments.

* Remove custom emoji from text before language detection
Akihiko Odaki 48d77ea1eb Fix filterable_languages method of SettingsHelper ()
Eugen Rochko 1caf11ddcc Fix language filter codes () 
* Fix language filter codes

CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)

* Fix zh name
Eugen Rochko e1fcad34a9 Fix length validator counting things that look like URIs like URLs () 
URI.extract is too strong, not limited to URLs, matched real text.
Same issue was present in language detector.
Matt Jankowski 022008a2a6 Language detection defaults to nil () 
* Default to nil for statuses.language

* Language detection defaults to nil instead of instance UI default
Matt Jankowski d010e270e6 Remove usernames and hashtags from language detection () 
* Add failing specs for hashtag and username extraction in language detector

* Remove usernames and hashtags from text before language detection

* Handle multiple instances of special case, and reduce whitespace
Akihiko Odaki d5cabfe5c6 Use CLD3 () 
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
Matt Jankowski 7a7d12d27f Delegate Account#user_locale method and allow nil ()
Matt Jankowski 8c5ad23b24 Language improvements, replace whatlanguage with CLD () 
* add failing en specs

* add cld2 gem

* Replace WhatLanguage with CLD
Matt Jankowski 629d35e6f5 [WIP] Html lang on statuses () 
* Add html lang attributes around statuses

* Remove urls from language detection
Matt Jankowski 297c11dba2 Language detection refactor () 
* Extract detect_language to separate class

* Use default locale, not just en

* Add spec to confirm that whatlanguage cant identify empty string

* Allow account locale to override default in language detector

* PostStatusService supplies an account to detect language