Skip to main content

Content Moderation & Reporting — Technical Reference

← Back to Content Moderation & Reporting

Where this lives​

Backend

Frontend

  • apps/frontend-nextjs/src/components/PostOptionsMenu.tsx — real report modal with a reasons list, calls createContentReport; rendered from PostCard.tsx, PostModal.tsx, and HomePage.tsx (this doc previously said no frontend report button existed anywhere — corrected)
  • apps/frontend-admin/src/app/moderation/page.tsx — report review queue: lists/filters via adminGetReports, approve/reject/escalate via adminReviewReport, and bulk actions via adminBulkReviewReports (this doc previously said there was no admin dashboard for reports — corrected)
  • apps/frontend-admin/src/app/moderation/flagged/page.tsx — flagged-content queue and detail panel: adminGetFlaggedContent, adminGetContentDetails, adminGetContentModerationStats, plus approve/reject/remove/restore/flag/unflag/warn/bulk-remove actions (this doc previously said there was no frontend for the flagged-content queue — corrected)
  • apps/frontend-admin/src/app/moderation/rules/page.tsx — auto-moderation rule CRUD UI: adminGetAutoModerationRules, adminCreateAutoModerationRule, adminUpdateAutoModerationRule, adminDeleteAutoModerationRule (this doc previously said the auto-moderation rules engine wasn't reachable via the API at all — corrected)

Technical implementation checklist​

  • createContentReport / myReports — resolvers wired in content-report.resolver.js; PostOptionsMenu.tsx calls createContentReport (corrected — see above)
  • adminGetReports / adminReviewReport / adminBulkReviewReports — resolvers wired; now wired to the frontend-admin /moderation page (Aprobar/Rechazar/Escalar buttons all route through adminReviewReport's action param) — this doc previously said no admin dashboard existed for this; corrected. adminEscalateReport and adminDismissReport have been removed from the schema entirely (no longer in content-report.type.js or the resolver) — escalation and dismissal now happen only via adminReviewReport with action: ESCALATE / action: REJECT
  • NSFW detection (nsfw-detection.service.js) — fully automated; genuinely wired into the upload pipeline in upload.service.js and runs on every image/video upload; no dedicated frontend needed for this one
  • adminWarnUser — resolver wired in graphql/resolvers/admin/user-moderation.resolver.js + data-access-services/admin/user-warnings.access-service.js; backend ready, no frontend found
  • adminGetFlaggedContent / adminApproveContent / adminRemoveContent / etc. — resolvers wired; now wired to the frontend-admin /moderation/flagged page (list, detail panel, and all content actions) — this doc previously said no frontend existed for this; corrected
  • adminAddContentWarning — resolver wired and now wired to the /moderation/flagged page's "Advertencia" action (previously said no frontend found; corrected)
  • adminRemoveContentWarning — resolver wired; backend ready, no frontend found
  • Auto-moderation rules engine — now real and reachable (previously said not wired to a resolver; corrected). adminGetAutoModerationRules / adminCreateAutoModerationRule / adminUpdateAutoModerationRule / adminDeleteAutoModerationRule are implemented in content-moderation.manager.js against a real auto_moderation_rule DB table, and wired to a full CRUD UI at /moderation/rules. Rules are also actually enforced by services/auto-moderation.service.js, called from post.manager.js on every post creation.

Reportable content types​

enum ContentType { post  comment  message  user }

article/blast/tale used to be listed here too, but never had a backing database model - they were dead stub access-services (data-access-services/admin/{blast,tale,article}.access-service.js, always returning empty results) kept alive only so admin moderation code didn't crash on them. Both the stubs and the enum values have been removed. Reporting content whose media is a single video (a "Clip") already uses contentType: post, since Clips are just Post rows with type: 'clip' - see Clips.

Report reasons​

enum ReportReason {
  spam · nudity · hate_speech · violence · harassment
  false_information · scam · intellectual_property
  self_harm · terrorism · other
}

Report statuses​

enum ReportStatus { pending  reviewing  resolved  dismissed }

Submitting a report​

createContentReport submits a user report for any piece of content. The contentType identifies what is being reported; contentId is that item's database ID. The reason and optional description are stored for admin review. The mutation returns the report's ID and initial status (pending).

mutation CreateContentReport($input: ContentReportCreateInput!) {
  createContentReport(input: $input) { id contentType reason status createdAt }
}

User queries​

myReports lets a user review the reports they have submitted and their current statuses — useful for a "Your reports" settings screen.

contentReportCount returns the total number of reports on a piece of content. Admins use this to prioritize review, and it can be shown in the flagged-content queue.

query MyReports($limit: Int) {
  myReports(limit: $limit) { id contentType reason status createdAt }
}

query ContentReportCount($contentType: ContentType!, $contentId: ID!) {
  contentReportCount(contentType: $contentType, contentId: $contentId)
}

Admin queries​

adminGetReports is the main moderation queue. Filter by status: pending to show unreviewed reports; filter by contentType to focus on a specific kind of content. Results include who filed the report and when it was last reviewed.

adminGetReportStats returns aggregate counters by type and reason — used to build the moderation dashboard overview.

adminGetReportDetails fetches a single report with full context: the original content, the reporter's history (how many reports they've filed), and other similar reports on the same content. Use this when an admin opens a report for review.

query AdminGetReports($status: ReportStatus, $contentType: ContentType, $limit: Int) {
  adminGetReports(status: $status, contentType: $contentType, limit: $limit) {
    reports { id contentType reason status reporter { username } reviewedAt }
    total
  }
}

query AdminGetReportStats {
  adminGetReportStats {
    totalReports pendingReports resolvedReports dismissedReports
    reportsByType { post comment message user }
    reportsByReason { spam nudity harassment violence }
  }
}

query AdminGetReportDetails($reportId: ID!) {
  adminGetReportDetails(reportId: $reportId) {
    report { id reason status reviewNotes }
    content
    reporterHistory { id reason createdAt }
    similarReports { id reason }
  }
}

Admin actions​

adminReviewReport resolves a single report. The action field accepts values like approve, reject, or remove — it also covers escalation (action: ESCALATE, sets status to reviewing) and dismissal (action: REJECT, sets status to dismissed); dedicated adminEscalateReport/adminDismissReport mutations existed previously but have been removed from the schema (the underlying content-report.manager.js methods escalateReport()/dismissReport() still exist but are now unreachable dead code). Pass removeContent: true to also delete the reported item. adminBulkReviewReports applies the same action to multiple reports at once — useful for clearing spam batches.

mutation AdminReviewReport($reportId: ID!, $action: String!, $notes: String, $removeContent: Boolean) {
  adminReviewReport(reportId: $reportId, action: $action, notes: $notes, removeContent: $removeContent) {
    success report { status reviewNotes }
  }
}

mutation AdminBulkReviewReports($reportIds: [ID!]!, $action: String!) {
  adminBulkReviewReports(reportIds: $reportIds, action: $action) { success }
}

Comment controls (admin)​

adminDisableComments turns off commenting on a specific post — useful when it's generating harassment but shouldn't be removed. adminEnableComments reverses it.

mutation AdminDisableComments($contentType: String!, $contentId: ID!, $reason: String) {
  adminDisableComments(contentType: $contentType, contentId: $contentId, reason: $reason) { success }
}

mutation AdminEnableComments($contentType: String!, $contentId: ID!) {
  adminEnableComments(contentType: $contentType, contentId: $contentId) { success }
}

Flagged content queue​

adminGetFlaggedContent returns a unified queue of all flagged content regardless of report source (user report, auto-moderation, NSFW score). Filter by contentType or status. Each item includes a preview (text snippet/thumbnail, built by content-preview.service.js) and the name of the reviewing moderator — FlaggedContent itself has no nsfwScore/toxicityScore fields; those live on ContentModerationDetails below.

adminGetContentDetails fetches the full context for a single piece of content: its raw data, AI scores, all associated reports, and the full moderation history. This is the detail view an admin sees when opening a flagged item. nsfwScore is a real field resolver that looks up the worst-case NSFW confidence score recorded for the content (posts only, from nsfw-detection.manager.js's scan on post creation). toxicityScore now also has a real field resolver, fixed this session: text toxicity is only ever persisted via NsfwCommentScore.toxicityScore, written for comments only (see database/models/NsfwCommentScore.js) — there's no equivalent toxicity model/column for posts — so the resolver returns that value for contentType: COMMENT and null for every other content type. That null for posts is intentional (no signal exists to return), not a bug. moderationHistory (via adminGetContentModerationHistory) now returns real data with the correct shape — fixed this session: the manager previously mapped log rows to {id, action, moderatorId, moderatorType, previousStatus, newStatus, metadata, createdAt}, a shape that doesn't exist on the schema's ModerationAction type at all (actionType/reason/performedBy/performedAt/notes), so every real call threw a GraphQL serialization error on the non-null actionType/performedBy/performedAt fields. It's now mapped to the real field names, with entries whose moderatorId can't be resolved to a real admin (e.g. 'system' auto-moderation) dropped rather than violating performedBy's non-null constraint.

adminGetContentModerationStats returns period-over-period moderation metrics for the dashboard — how much was auto-moderated vs. manually reviewed, and how much was approved vs. removed.

query AdminFlaggedContent($contentType: String, $status: String, $limit: Int, $offset: Int) {
  adminGetFlaggedContent(contentType: $contentType, status: $status, limit: $limit, offset: $offset) {
    total content {
      id contentType contentId status flagReason flagCategory flaggedAt
      author { username }
      reviewedBy { username }
      moderationAction moderationNotes
    }
  }
}

query AdminContentDetails($contentType: String!, $contentId: ID!) {
  adminGetContentDetails(contentType: $contentType, contentId: $contentId) {
    contentType contentId content status
    nsfwScore toxicityScore
    author { username }
    reports { id reason status }
    moderationHistory { actionType reason performedAt performedBy { username } }
  }
}

query AdminModerationStats($period: String) {
  adminGetContentModerationStats(period: $period) {
    totalFlagged pendingReview approvedContent removedContent
    autoModerated manualModerated
  }
}

Admin content actions​

adminApproveContent marks a flagged item as safe and clears it from the review queue. adminRejectContent flags it for removal but doesn't delete it yet — useful for a two-step confirmation flow.

adminRemoveContent permanently deletes the content. Pass notifyUser: true to send the author an in-app notification explaining the removal. adminRestoreContent reverses a previous removal.

adminBulkRemoveContent deletes multiple items in one call — each item in items is a JSON object with contentType and contentId.

adminFlagContent manually adds a content item to the review queue (e.g. an admin spotted a problem through normal browsing). adminUnflagContent removes it.

mutation AdminApproveContent($input: ModerationActionInput!) { adminApproveContent(input: $input) { success } }
mutation AdminRejectContent($input: ModerationActionInput!)  { adminRejectContent(input: $input) { success } }

# Permanently delete content; optionally notify the author
mutation AdminRemoveContent($contentType: String!, $contentId: ID!, $reason: String!, $notifyUser: Boolean) {
  adminRemoveContent(contentType: $contentType, contentId: $contentId, reason: $reason, notifyUser: $notifyUser) { success }
}

# Restore previously removed content
mutation AdminRestoreContent($contentType: String!, $contentId: ID!, $reason: String!) {
  adminRestoreContent(contentType: $contentType, contentId: $contentId, reason: $reason) { success }
}

# Bulk-delete multiple items (items: [{ contentType, contentId }])
mutation AdminBulkRemoveContent($items: [JSON!]!, $reason: String!) {
  adminBulkRemoveContent(items: $items, reason: $reason) { success }
}

mutation AdminFlagContent($contentType: String!, $contentId: ID!, $reason: String!, $priority: String) {
  adminFlagContent(contentType: $contentType, contentId: $contentId, reason: $reason, priority: $priority) { success }
}
mutation AdminUnflagContent($contentType: String!, $contentId: ID!, $reason: String) {
  adminUnflagContent(contentType: $contentType, contentId: $contentId, reason: $reason) { success }
}

Content warnings​

Admins can attach severity-tagged warnings to content (e.g. "sensitive content", "graphic imagery"). The warning is surfaced to users before they see the content. severity accepts values like low, medium, high.

adminRemoveContentWarning clears whichever warning is currently active on a piece of content. Investigated this session: this section previously described the resolver as "ignoring" a warningId argument. That doesn't reproduce — the schema declares no warningId argument at all, and never did; it's adminRemoveContentWarning(contentType: String!, contentId: ID!): ModerationActionResponse!. This is by design, not a bug: a content item carries at most one active warning at a time (flat columns on the row, not a per-warning table — see database/migrations/20260730180000-add-content-warning-to-post.js), so there's no per-warning ID to target in the first place.

mutation AddContentWarning($contentType: String!, $contentId: ID!, $warning: String!, $severity: String) {
  adminAddContentWarning(contentType: $contentType, contentId: $contentId, warning: $warning, severity: $severity) { success }
}
mutation RemoveContentWarning($contentType: String!, $contentId: ID!) {
  adminRemoveContentWarning(contentType: $contentType, contentId: $contentId) { success }
}

Auto-moderation rules​

Auto-moderation rules run server-side and apply actions (remove, flag, warn) automatically based on JSON conditions (e.g. nsfwScore > 0.9) and JSON actions (e.g. { "action": "remove" }). This reduces moderator workload for clear-cut violations. This is a real, working feature: CRUD is backed by a real auto_moderation_rule DB table (data-access-services/admin/auto-moderation-rule.access-service.js, model AutoModerationRule.js), and rules are actually enforced by services/auto-moderation.service.js — a rule engine matching keyword/regex/nsfw_score/report_threshold rule types, called (best-effort, non-blocking) from post.manager.js on every post creation. There's a full CRUD UI at apps/frontend-admin/src/app/moderation/rules/page.tsx.

adminGetAutoModerationRules lists all configured rules with their active/inactive state. adminCreateAutoModerationRule adds a new rule (super_admin only). adminUpdateAutoModerationRule edits an existing one (e.g. to adjust the threshold; super_admin only). adminDeleteAutoModerationRule permanently removes the rule (super_admin only).

query AutoModerationRules { adminGetAutoModerationRules { id name ruleType conditions actions isActive } }

mutation CreateAutoModerationRule($input: AutoModerationRuleInput!) {
  adminCreateAutoModerationRule(input: $input) { id name ruleType isActive }
}
mutation UpdateAutoModerationRule($id: ID!, $input: AutoModerationRuleInput!) {
  adminUpdateAutoModerationRule(id: $id, input: $input) { id isActive }
}
mutation DeleteAutoModerationRule($id: ID!) {
  adminDeleteAutoModerationRule(id: $id) { success }
}

AutoModerationRule fields: name, description, ruleType, conditions (JSON), actions (JSON), isActive.

Automatic NSFW detection​

The backend integrates Google Cloud AI to automatically scan uploaded content:

ModulePurpose
utils/apis-google/gc-vision.jsImage NSFW analysis via Cloud Vision
utils/apis-google/gc-streaming-video-intelligence.jsVideo NSFW analysis via Video Intelligence

Scores (nsfwScore, toxicityScore) are attached to content in ContentModerationDetails and can trigger auto-moderation rules or manual review.