Commit Graph

705 Commits (9bf648ac911be4729283ef5e34f7f4c9bd86c97e)

Author SHA1 Message Date
Johnny dc7ec8a8ad feat: allow setting custom timestamps when creating memos and comments
Allow API users to set custom create_time, update_time, and display_time
when creating memos and comments. This enables importing historical data
with accurate timestamps.

Changes:
- Update proto definitions: change create_time and update_time from
  OUTPUT_ONLY to OPTIONAL to allow setting on creation
- Modify CreateMemo service to handle custom timestamps from request
- Update database drivers (SQLite, MySQL, PostgreSQL) to support
  inserting custom timestamps when provided
- Add comprehensive test coverage for custom timestamp functionality
- Maintain backward compatibility: auto-generated timestamps still
  work when custom values are not provided
- Fix golangci-lint issues in plugin/filter (godot and revive)

Fixes #5483
5 months ago
Johnny cbf46a2988 feat(filter): add CEL list comprehension support for tag filtering
Add support for CEL exists() comprehension with startsWith, endsWith, and
contains predicates to enable powerful tag filtering patterns.

Features:
- tags.exists(t, t.startsWith("prefix")) - Match tags by prefix
- tags.exists(t, t.endsWith("suffix")) - Match tags by suffix
- tags.exists(t, t.contains("substring")) - Match tags by substring
- Negation: !tags.exists(...) to exclude matching tags
- Works with all operators (AND, OR, NOT) and other filters

Implementation:
- Added ListComprehensionCondition IR type for comprehension expressions
- Parser detects exists() macro and extracts predicates
- Renderer generates optimized SQL for SQLite, MySQL, PostgreSQL
- Proper NULL/empty array handling across all database dialects
- Helper functions reduce code duplication

Design decisions:
- Only exists() supported (all() rejected at parse time with clear error)
- Only simple predicates (matches() excluded to avoid regex complexity)
- Fail-fast validation with helpful error messages

Tests:
- Comprehensive test suite covering all predicates and edge cases
- Tests for NULL/empty arrays, combined filters, negation
- Real-world use case test for Issue #5480 (archive workflow)
- All tests pass on SQLite, MySQL, PostgreSQL

Closes #5480
5 months ago
Johnny 4bd0f29ee5 fix: resolve linter warning and cache race condition
- Suppress revive redundant-test-main-exit warning with //nolint
- Fix data race in cache.Clear() by using Delete() instead of map replacement
- Both changes maintain correct behavior while fixing CI failures
5 months ago
Johnny db57f4456a fix: resolve data races in store tests with atomic operations
Fix all data races detected by -race flag in parallel store tests.

Problems fixed:
1. TestMain not propagating exit code
   - Was: m.Run(); return
   - Now: os.Exit(m.Run())

2. Data races on global DSN variables
   - mysqlBaseDSN and postgresBaseDSN written in sync.Once
   - Read outside Once without synchronization
   - Race detector: write/read without happens-before relationship

3. Data races on container pointers
   - mysqlContainer, postgresContainer, testDockerNetwork
   - Same pattern: write in Once, read in cleanup

Solution: Use atomic operations
- atomic.Value for DSN strings (Store/Load)
- atomic.Pointer for container pointers (Store/Load)
- Provides proper memory synchronization
- Race-free reads and writes

Why sync.Once alone wasn't enough:
- sync.Once guarantees function runs once
- Does NOT provide memory barrier for variables written inside
- Reads outside Once have no synchronization with writes
- Race detector correctly flags this as violation

Technical details:
- atomic.Value.Store() provides release semantics
- atomic.Value.Load() provides acquire semantics
- Guarantees happens-before relationship per Go memory model
- All 159 parallel tests can safely access globals

Impact:
- Tests now pass with -race flag
- No performance degradation (atomic ops are fast)
- Maintains parallel execution benefits (8-10x speedup)
- Proper Go memory model compliance

Related: Issues #2, #3 from race analysis
5 months ago
Johnny 411e8fc5b0 perf: enable parallel execution for all store tests
Add t.Parallel() to all 159 test functions in store/test/

Benefits:
- Tests run in parallel (8-16 concurrent on typical CI)
- Each test already has isolated database (safe to parallelize)
- No shared state between tests
- Go test runner handles synchronization

Expected performance:
- SQLite tests: 4-6min → 30-45sec (87% faster)
- MySQL tests: 6-8min → 45-60sec (88% faster)
- Better CPU utilization (10-15% → 80-95%)

Why it's safe:
- NewTestingStore() creates isolated DB per test
- No global state mutations
- Each test uses t.TempDir() for file isolation
- Container-based tests use unique database names

Impact on CI workflow:
- Backend tests workflow: 4-6min → 1-2min total
- Store test group: 3-4min → 20-30sec
- 8-10x speedup from parallelization alone
5 months ago
Johnny c45a59549a fix: replace os.Exit with panic for clearer error handling 5 months ago
Johnny 69b62cccdb test: optimize store tests performance by reusing docker image and reducing build context 5 months ago
Johnny f58533003b fix: clean up memo_relation and attachments when deleting memo
Fixes #5472

Move cleanup logic to store.DeleteMemo to ensure data consistency:
- Delete memo_relation records where memo is source (MemoID) or target (RelatedMemoID)
- Delete attachments linked to the memo (including S3/local files)

This prevents stale COMMENT records in memo_relation after deleting
a memo that has comments.
5 months ago
Johnny 6899c2f66a chore: fix golangci-lint 5 months ago
Johnny 31f634b71a chore: add more tests 5 months ago
Johnny 1e8505bfeb chore: fix tests 5 months ago
Johnny af6a0726d8 chore: drop redundant indexes in sqlite migration 5 months ago
Johnny 1985205dc2 chore: remove memo_organizer table 5 months ago
Johnny f4cf2e9559 test: improve migration tests stability and maintainability
- Fix linting issues and address testcontainers deprecation in store/test/containers.go
- Extract MemosStartupWaitStrategy for consistent container health checks
- Refactor migrator_test.go to use a consolidated testMigration helper, reducing duplication
- Add store/test/Dockerfile for optimized local test image builds
5 months ago
Johnny 14fb38f375 fix: attachment table name 5 months ago
Johnny c29c1d3e1f fix: seed data 5 months ago
Johnny cc9a214be8 revert: revert system_setting to instance_setting rename changes
Reverts only the system_setting → instance_setting rename related changes from commit d326c71.
Keeps the resource → attachment rename changes intact.

- Reverts table name back to system_setting in all database drivers (MySQL, PostgreSQL, SQLite)
- Removes migration files for the system_setting rename
- Reverts LATEST.sql files to use system_setting table
5 months ago
Johnny e75862de31 chore: fix tests 5 months ago
Johnny 64b487d4af chore: fix seed data 5 months ago
Johnny d326c71078 refactor(db): rename tables for clarity - resource→attachment, system_setting→instance_setting 5 months ago
Johnny d68ca84832 fix: delete unused attachments by using filter 5 months ago
Johnny bd02de9895
chore: add store tests (#5397) 6 months ago
Johnny 5d677828a6 refactor: remove NSFW content settings from instance configuration 6 months ago
Johnny 78aa41336a feat: implement attachment filtering functionality 6 months ago
Steven d0c3908168 refactor: remove deprecated Sessions and AccessTokens settings
- Remove ListSessions and RevokeSession RPC endpoints
- Remove Session message and SessionsSetting from UserSetting
- Remove ACCESS_TOKENS key and AccessTokensSetting
- Update references to use RefreshTokensUserSetting with its own ClientInfo
- Remove UserSessionsSection frontend component
- Clean up user store to remove session and access token settings
- Regenerate protobuf files

The system now uses:
- REFRESH_TOKENS for session management with sliding expiration
- PERSONAL_ACCESS_TOKENS for long-lived API tokens
6 months ago
Johnny 7932f6d0d0
refactor: user auth improvements (#5360) 6 months ago
Johnny 310590b278 chore: fix golang checks 6 months ago
Johnny 40e129b8af refactor(auth): streamline session authentication and cookie handling 6 months ago
Steven 0610257562 refactor(store): remove deprecated migration_history table and backward compatibility code
Complete removal of migration_history system in favor of instance_setting based schema versioning.

Changes:
- Remove migration_history table creation from all LATEST.sql files
- Delete all migration_history model and implementation files (~300 lines)
- Remove FindMigrationHistoryList and UpsertMigrationHistory from Driver interface
- Replace complex backward compatibility functions with simple version check
- Update health check to use instance_setting instead of migration_history
- Simplify checkMinimumUpgradeVersion to detect pre-v0.22 installations

Breaking change:
Users on versions < v0.22.0 (May 2024) must upgrade to v0.25.x first before upgrading to this version.
Clear error message with upgrade instructions will be shown for old installations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
7 months ago
Steven d1492007ab fix(store): filter inbox notifications by message type at database level
Add MessageType filter to FindInbox to exclude legacy VERSION_UPDATE
notifications from inbox queries. This resolves the issue where users
saw notification counts but no items displayed, as VERSION_UPDATE
entries cannot be rendered in the new UserNotification API.

Changes:
- Add MessageType field to FindInbox struct for database-level filtering
- Implement JSON extraction filters in SQLite, MySQL, and PostgreSQL drivers
- Update ListUserNotifications to filter MEMO_COMMENT type at store level

This approach improves performance by filtering at the database rather
than in application code, reducing unnecessary data transfer for users
with many legacy inbox entries.

Fixes #5278

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
7 months ago
Steven 07a030ddfd fix(postgres): update tag filtering SQL to ensure proper type casting for LIKE comparisons 7 months ago
Steven 50f49fc00d chore: update demo data 7 months ago
Steven 424f11f227 fix(store): fix PostgreSQL tag filtering type inference error
Resolves issue where tag filtering in PostgreSQL databases failed with "operator does not exist: jsonb ~~ unknown" error. The hierarchical tag filtering feature introduced in commit 5e47f25b generated SQL with implicit type placeholders that PostgreSQL couldn't infer.

The fix explicitly casts the LIKE comparison placeholder to text (::text) in the PostgreSQL dialect, ensuring proper type resolution for the query parameter.

Fixes #5275

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
7 months ago
Steven d1b2c0308b chore: add demo user access token for testing purposes 7 months ago
boojack 21d31e3609
fix(security): implement security review recommendations (#5228)
Co-authored-by: Claude <noreply@anthropic.com>
7 months ago
Steven 4c1d1c70d1 refactor: rename workspace to instance throughout codebase
Remove work-related terminology by renaming "workspace" to "instance"
across the entire application. This change better reflects that Memos
is a self-hosted tool suitable for personal and non-work use cases.

Breaking Changes:
- API endpoints: /api/v1/workspace/* → /api/v1/instance/*
- gRPC service: WorkspaceService → InstanceService
- Proto types: WorkspaceSetting → InstanceSetting
- Frontend translation keys: workspace-section → instance-section

Backend Changes:
- Renamed proto definitions and regenerated code
- Updated all store layer methods and database drivers
- Renamed service implementations and API handlers
- Updated cache from workspaceSettingCache to instanceSettingCache

Frontend Changes:
- Renamed service client: workspaceServiceClient → instanceServiceClient
- Updated all React components and state management
- Refactored stores: workspace.ts → instance.ts
- Updated all 32 locale translation files

All tests pass and both backend and frontend build successfully.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
7 months ago
Steven bc1550e926 refactor(api): migrate inbox functionality to user notifications
- Remove standalone InboxService and move functionality to UserService
- Rename inbox to user notifications for better API consistency
- Add ListUserNotifications, UpdateUserNotification, DeleteUserNotification methods
- Update frontend components to use new notification endpoints
- Update store layer to support new notification model

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
8 months ago
Steven 5e47f25bf5 feat(store): add hierarchical tag filtering support
Tag filters now support hierarchical matching where searching for a tag (e.g., "book") will match both the exact tag and any tags with that prefix (e.g., "book/fiction", "book/non-fiction"). This applies across all database backends (SQLite, MySQL, PostgreSQL) with corresponding test updates.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
8 months ago
Steven f635d85bcf chore: fix reactions seed data
- Add comprehensive inline documentation for auth services
- Document session-based and token-based authentication flows
- Clarify authentication priority and validation logic
- Add detailed comments for JWT token structure and claims
- Fix reactions seed data to use memo UIDs instead of numeric IDs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
8 months ago
Claude f241b590a2 chore: update demo data 8 months ago
Steven e0b1153269 fix(web): resolve MobX observable reactivity issue in filter computation
Fixes filtering functionality that was broken due to improper use of
useMemo with MobX observables. The issue occurred because useMemo's
dependency array uses reference equality, but MobX observable arrays
are mutated in place (reference doesn't change when items are added/removed).

Changes:
- Remove useMemo from filter computation in Home, UserProfile, and Archived pages
- Calculate filters directly in render since components are already MobX observers
- Fix typo: memoFitler -> memoFilter in Archived.tsx

This ensures filters are recalculated whenever memoFilterStore.filters changes,
making tag clicks and other filter interactions work correctly.

Fixes #5189

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
8 months ago
Steven 46ce0bc62e fix(store): correct PostgreSQL placeholder generation in IN clauses
Fixes a regression introduced in v0.25.2 where PostgreSQL IN clause
placeholders were not properly incremented, causing all parameters to
use the same placeholder index (e.g., $1, $1, $1 instead of $1, $2, $3).

This bug affected:
- ListReactions (ContentIDList) - caused "failed to list reactions" errors
- ListAttachments (MemoIDList)
- ListMemos (IDList and UIDList)

The fix combines placeholder generation and argument appending into a
single loop to ensure proper incrementing.

Fixes #5188

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
8 months ago
Claude e35f16306e feat: add id as final tie-breaker in ListMemos 8 months ago
Steven 2d4361d5fe chore: update memo content formatting and enhance link styling 8 months ago
Steven d8f7a4e739 chore: tweak demo data 8 months ago
Steven b19e736f10 chore: tweak demo data 8 months ago
Steven b4ea7d843f feat: enhance memo sorting functionality to support multiple fields 8 months ago
Steven 95de5cc700 refactor: update migration history methods 8 months ago
Johnny bc7decf642 refactor: remove unused constants 8 months ago
Copilot b685ffacdf refactor: memo filter
- Updated memo and reaction filtering logic to use a unified engine for compiling filter expressions into SQL statements.
- Removed redundant filter parsing and conversion code from ListMemoRelations, ListReactions, and ListAttachments methods.
- Introduced IDList and UIDList fields in FindMemo and FindReaction structs to support filtering by multiple IDs.
- Removed old filter test files for reactions and attachments, as the filtering logic has been centralized.
- Updated tests for memo filtering to reflect the new SQL statement compilation approach.
- Ensured that unsupported user filters return an error in ListUsers method.
8 months ago
Steven c3d4f8e9d1 feat: implement user-specific SQL converter for filtering in user service 9 months ago
Johnny 7cc2df9254 chore: fix linter 10 months ago
varsnotwars 4eb5b67baf
feat: attachments by id (#5008) 10 months ago
varsnotwars f4bdfa28a0
feat: filter/method for reactions by content_id (#4969) 10 months ago
Colin Holzman 8319516d1a
fix: boolean filters (#4966) 10 months ago
johnnyjoy 506b477d50 fix: get user by username 11 months ago
varsnotwars fa2fa8a5d7
refactor: remove call to db for parent memo name (#4947) 11 months ago
johnnyjoy 34fb3f1514 chore: fix linter 11 months ago
Colin Holzman aae7ec8d1f
fix: calendar filter (#4942) 11 months ago
Maximilian Krauß 4d6042e35f
fix(tags): ensure JSON array elements are properly formatted in SQL queries (#4944) 11 months ago
Andrea Marchetta 71464779dd
fix: change itemCount into an Int64 (#4945) 11 months ago
johnnyjoy ed23cbc011 refactor: memo filter 11 months ago
Steven 1a75d19a89 fix: memo filter for sqlite 11 months ago
johnnyjoy 7098721617 chore: fix linter 11 months ago
johnnyjoy e24f92b719 fix: tests 11 months ago
johnnyjoy b55904a428 feat: support more filter factors 11 months ago
johnnyjoy 6d9770b9c8 chore: move filter to filters 11 months ago
johnnyjoy 7481fe10bf chore: remove order by pinned 11 months ago
Johnny 40130350c5 fix: args offset for postgres 11 months ago
Steven 834f92f67c chore: fix linter 11 months ago
Steven 0398df1d21 chore: update migrator comments 11 months ago
Johnny 3e24f8cf62
fix: seed data 12 months ago
Steven 42d1650c6d chore: tweak auth service 12 months ago
Steven 6e1b01cb68 chore: bump version 12 months ago
johnnyjoy d6a75bba4c refactor: webhook service 12 months ago
jinjingroad 03399a6007
refactor: use the built-in max/min to simplify the code (#4781)
Signed-off-by: jinjingroad <jinjingroad@sina.com>
12 months ago
Johnny e6e460493c refactor: general user setting 12 months ago
Johnny 9b15936873 refactor: clean unused fields 12 months ago
Johnny 778a5eb184 refactor: memo filter 12 months ago
Johnny 77b7fc4441 feat: implement user session 12 months ago
Steven 9972a77d9e refactor: memo service 1 year ago
Steven a4920d464b refactor: attachment service part2 1 year ago
Steven 8e8e246ab2 chore: add server tests 1 year ago
Johnny 38b6c3afb9 chore: fix linter 1 year ago
Johnny 3fd29f6493 refactor: schema migrator 1 year ago
Johnny b942643785 chore: tweak seed data 1 year ago
Steven f1b365f928 refactor: clean packages 1 year ago
Steven de3e55c2e6 feat: support `now()` time functions 1 year ago
Steven 8520e30721 fix: handle type assertion safely 1 year ago
Steven ef6f80d925 chore: fix linter 1 year ago
Steven 81ae42518e chore: fix linter 1 year ago
Steven ad2c5f0d05 refactor: store cache 1 year ago
Steven b89d8f5342 feat: implement hasTaskList filter 1 year ago
johnnyjoy f9e07a2245 feat: support update user's role 1 year ago
Steven 1f79ee575b fix: create user with avatar url for sqlite 1 year ago
johnnyjoy 2a92baf52c fix: filter args of postgres 1 year ago
Johnny b94682dc4f
chore: fix linter 1 year ago
Johnny 18b9b9d18f
chore: implement memo filter in list memo relations 1 year ago
Johnny 08f9b18ced
fix: list memo relations 1 year ago
Johnny 3cc0e255ce fix: memo filter for postgres 1 year ago