第56章 NodeBB 数据库结构解析:Redis 键值模型与 MongoDB 映射
第56章 数据库结构
NodeBB 的数据存储基于 redis 的几种结构(详见这里):hash(哈希)、set(集合)、sorted set(有序集合)和 list(列表)。
hash 类似于 javascript 对象;set 是不重复值的集合;sorted set 和 set 一样,但多了一个 score 字段用于排序;list 则类似 JS 中的数组。
比如每个用户的数据都存在一个对象里,键名为 `user:
NodeBB 用以下键名指代各种 id:
- uid - 用户 id `user:
凡是你看到 `user:
要获取用户 100 的数据,可以在 redis 中执行 `hgetall user:100`,或在 mongodb 中执行 `db.objects.findOne({_key: "user:100"})`。
下面是 mongodb 的输出:
``` > db.objects.findOne({_key:"user:100"}) { "_id" : ObjectId("555b9afa65190fe2123df8de"), "_key" : "user:100", "username" : "KwVLmsBU", "userslug" : "kwvlmsbu", "email" : "[email protected]", "joindate" : 1432066810034, "picture" : "", "fullname" : "", "location" : "", "birthday" : "", "website" : "", "signature" : "", "uploadedpicture" : "", "profileviews" : 0, "reputation" : 0, "postcount" : 0, "topiccount" : 0, "lastposttime" : 0, "banned" : 0, "status" : "online", "uid" : 100 } ```
帖子、话题、分类也是同样的结构,以下是几个最重要对象的示例输出。
全局对象(Global Object)—— 存储论坛中其他对象的计数器:
``` > db.objects.findOne({_key: "global"}) { "_id" : ObjectId("5c9be01d375afb4e6c577081"), "_key" : "global", "nextEid" : 183, "nextCid" : 26, "nextUid" : 23, "userCount" : 23, "nextTid" : 36, "topicCount" : 34, "nextPid" : 357, "postCount" : 355, "uniqueIPCount" : 149, "nextChatRoomId" : 10, "nextMid" : 55 } ```
配置对象(Config Object)—— 存储论坛全局设置:
``` > db.objects.findOne({_key: "config"}) { "_id" : ObjectId("5c9bdff1375afb4e6c577073"), "_key" : "config", "email:from" : "[email protected]", "adminReloginDuration" : 60, "allowAccountDelete" : 1, "allowGuestHandles" : 0, "allowMultipleBadges" : 0, "allowPrivateGroups" : 1, "allowProfileImageUploads" : 1, "allowTopicsThumbnail" : 0, "allowUserHomePage" : 0, "allowedFileExtensions" : "png,jpg,bmp", "autoDetectLang" : 1, "bookmarkThreshold" : 5, "categoryWatchState" : "watching", "chatDeleteDuration" : 5, "chatEditDuration" : 5, "chatMessageDelay" : 200, "defaultLang" : "en-US", "digestHour" : 17, "disableChat" : 0, "disableEmailSubscriptions" : 0, "disableRecentCategoryFilter" : 0, "disableSignatures" : 0, "downvote:disabled" : 0, "email:disableEdit" : 0, "emailConfirmInterval" : 10, "enablePostHistory" : 1, "eventLoopCheckEnabled" : 1, "eventLoopInterval" : 500, "eventLoopLagThreshold" : 100, "feeds:disableSitemap" : 1, "gdpr_enabled" : 0, "hideFullname" : 0, "hsts-enabled" : 0, "hsts-maxage" : 31536000, "hsts-preload" : 0, "hsts-subdomains" : 0, "initialPostDelay" : 10, "inviteExpiration" : 7, "lockoutDuration" : 60, "loginAttempts" : 5, "loginDays" : 14, "loginSeconds" : 0, "maintenanceMode" : 0, "maxPostsPerPage" : 20, "maxTopicsPerPage" : 20, "maximumAboutMeLength" : 1000, "maximumChatMessageLength" : 1000, "maximumCoverImageSize" : 2048, "maximumFileSize" : 2048, "maximumGroupNameLength" : 255, "maximumGroupTitleLength" : 40, "maximumInvites" : 0, "maximumPostLength" : 32767, "maximumProfileImageSize" : 256, "maximumRelatedTopics" : 0, "maximumSignatureLength" : 255, "maximumTagLength" : 15, "maximumTagsPerTopic" : 5, "maximumTitleLength" : 255, "maximumUsernameLength" : 16, "maximumUsersInChatRoom" : 0, "min:rep:aboutme" : 0, "min:rep:cover-picture" : 0, "min:rep:downvote" : 0, "min:rep:flag" : 0, "min:rep:profile-picture" : 0, "min:rep:signature" : 0, "min:rep:website" : 0, "minimumPasswordLength" : 6, "minimumPasswordStrength" : 0, "minimumPostLength" : 8, "minimumTagLength" : 3, "minimumTagsPerTopic" : 0, "minimumTitleLength" : 3, "minimumUsernameLength" : 2, "newbiePostDelay" : 120, "newbiePostDelayThreshold" : 3, "notificationType_follow" : "notification", "notificationType_group-invite" : "notification", "notificationType_mention" : "notification", "notificationType_new-chat" : "notification", "notificationType_new-post-flag" : "notification", "notificationType_new-register" : "notification", "notificationType_new-reply" : "notification", "notificationType_new-topic" : "notification", "notificationType_new-user-flag" : "notification", "notificationType_post-queue" : "notification", "notificationType_upvote" : "notification", "onlineCutoff" : 30, "passwordExpiryDays" : 0, "postCacheSize" : 10485760, "postDelay" : 10, "postDeleteDuration" : 0, "postEditDuration" : 0, "postsPerPage" : 20, "preventTopicDeleteAfterReplies" : 0, "profile:convertProfileImageToPNG" : 0, "profile:keepAllUserImages" : 0, "profileImageDimension" : 200, "registrationType" : "disabled", "rejectImageHeight" : 5000, "rejectImageWidth" : 5000, "reputation:disabled" : 0, "requireEmailConfirmation" : 0, "resizeImageQuality" : 80, "resizeImageWidth" : 760, "resizeImageWidthThreshold" : 2000, "showSiteTitle" : 0, "sitemapTopics" : 500, "teaserPost" : "last-post", "timeagoCutoff" : 30, "title" : "My Forums", "topicStaleDays" : 60, "topicThumbSize" : 120, "topicsPerPage" : 20, "unreadCutoff" : 2, "userSearchResultsPerPage" : 50, "username:disableEdit" : 0, "votesArePublic" : 0, "bootswatchSkin" : "", "theme:id" : "nodebb-theme-persona", "theme:src" : "", "theme:staticDir" : "", "theme:templates" : "", "theme:type" : "local", "customCSS" : "", "customHTML" : "", "customJS" : "", "enableLiveReload" : 1, "renderedCustomCSS" : "", "useCustomCSS" : 0, "useCustomHTML" : 0, "useCustomJS" : 1, "brand:favicon" : "", "brand:logo" : "", "brand:logo:alt" : "", "brand:logo:url" : "", "brand:touchIcon" : "", "browserTitle" : "", "description" : "", "keywords" : "", "og:image" : "", "outgoingLinks:whitelist" : "", "searchDefaultSortBy" : "relevance", "title:url" : "", "titleLayout" : "", "useOutgoingLinksPage" : 0, "homePageCustom" : "/home", "homePageRoute" : "categories", "homePageTitle" : "", "categoryTopicSort" : "oldest_to_newest", "composer:allowPluginHelp" : 1, "composer:customHelpText" : "", "composer:showHelpTab" : 1, "postQueue" : 1, "signatures:disableImages" : 0, "signatures:disableLinks" : 0, "topicPostSort" : "oldest_to_newest", "trackIpPerPost" : 0, "allowLoginWith" : "username-email", "dailyDigestFreq" : "off", "disableCustomUserSkins" : 0, "followTopicsOnCreate" : 0, "followTopicsOnReply" : 0, "hideEmail" : 0, "notificationType_group-request-membership" : "none", "openOutgoingLinksInNewTab" : 0, "password:disableEdit" : 0, "restrictChat" : 0, "showemail" : 0, "showfullname" : 0, "termsOfUse" : "", "topicSearchEnabled" : 0, "submitPluginUsage" : 0, "registrationApprovalType" : "admin-approval", "brand:emailLogo" : "\\assets\\uploads\\system\\site-logo-x50.png", "brand:logo:height" : 29, "brand:logo:width" : 32, "brand:emailLogo:height" : 50, "brand:emailLogo:width" : 155, "chat-incoming" : "Default | Water drop (high)", "chat-outgoing" : "Default | Deedle-dum", "notification" : "Default | Water drop (low)", "disableChatMessageEditing" : 0, "cookieConsentDismiss" : "", "cookieConsentEnabled" : 1, "cookieConsentLink" : "", "cookieConsentLinkUrl" : "", "cookieConsentMessage" : "", "cookieDomain" : "", "feeds:disableRSS" : 0, "robots:txt" : "", "backgroundColor" : "", "themeColor" : "", "title:short" : "", "groupsExemptFromPostQueue" : "[\"Global Moderators\",\"administrators\"]", "necroThreshold" : 7, "newbiePostEditDuration" : 3600, "postQueueReputationThreshold" : 2 } ```
分类对象(Category Object):
``` > db.objects.find({_key:"category:25"}).pretty() { "_id" : ObjectId("55d2c27ed03b9afd5af5e854"), "_key" : "category:25", "cid" : 25, "name" : "sub1", "description" : "", "icon" : "fa-comments", "bgColor" : "#AB4642", "color" : "#fff", "slug" : "25/sub1", "parentCid" : 23, "topic_count" : 0, "post_count" : 0, "disabled" : 0, "order" : 1, "link" : "", "numRecentReplies" : 1, "class" : "col-md-3 col-xs-6", "imageClass" : "auto" } ```
话题对象(Topic Object):
``` > db.objects.find({_key:"topic:2000"}).pretty() { "_id" : ObjectId("5547ae9d65190fe21227622c"), "_key" : "topic:2000", "tid" : 2000, "uid" : 668, "cid" : 2, "mainPid" : 15257, "title" : "Host Nodebb Free", "slug" : "2000/host-nodebb-free", "timestamp" : 1405433121145, "lastposttime" : 1421334586258, "postcount" : 34, "viewcount" : 1691, "locked" : 0, "deleted" : 0, "pinned" : 0, "teaserPid" : 25995, "upvotes": 0, "downvotes": 0 } ```
帖子对象(Post Object):
``` > db.objects.findOne({_key:"post:20000"}); { "_id" : ObjectId("5547af3f65190fe2122d0b3c"), "_key" : "post:20000", "edited" : 0, "pid" : 20000, "content" : "content of this post", "tid" : 2543, "timestamp" : 1412304172707, "deleted" : 0, "editor" : "", "uid" : 747, "toPid" : 19999, "upvotes" : 0, "downvotes": 0 } ```
插件设置对象 —— 插件的设置保存在 `settings:
``` > db.objects.findOne({_key: "settings:mentions"}) { "_id" : ObjectId("5e8c8db8ba5ceec316de0b92"), "_key" : "settings:mentions", "autofillGroups" : "off", "disableFollowedTopics" : "off", "disableGroupMentions" : "[]", "display" : "", "overrideIgnores" : "off" } ```
聊天消息对象(Chat Message Object):
``` > db.objects.findOne({_key: "message:1000"}) { "_id" : ObjectId("554537872b1e9e3c288d3b5e"), "_key" : "message:1000", "content" : "a chat message\n", "timestamp" : 1386347930646, "fromuid" : 2, "touid" : 243 } ```
通知对象(Notification Object)—— 根据创建方式不同,每个通知可能包含不同的字段:
``` > db.objects.findOne({_key: "notifications:chat_15142_3672"}) { "_id" : ObjectId("5ee0fe343dc567806d463a6f"), "_key" : "notifications:chat_15142_3672", "type" : "new-chat", "subject" : "subject", "bodyShort" : "short notification body", "bodyLong" : "long notification body", "nid" : "chat_15142_3672", "from" : 15142, "path" : "/chats/3672", "importance" : 5, "datetime" : 1591803444289 } ```
以上对象存储了论坛的全部数据。要以有序的方式检索数据,就要用到 sorted set——可以把它理解为索引,它有 value 和 score 两个字段。
以 `users:postcount` 这个 sorted set 为例:用户创建时会生成一个对象存储其数据,键名为 `user:
``` > db.objects.find({ _key:"users:postcount" }, { _id: 0, _key: 0 }).sort({ score: -1 }).pretty() { "value" : "2", "score" : 4121 } { "value" : "970", "score" : 2749 } { "value" : "3", "score" : 2190 } { "value" : "1", "score" : 1707 } { "value" : "598", "score" : 769 } { "value" : "11", "score" : 733 } { "value" : "477", "score" : 546 } { "value" : "587", "score" : 504 } { "value" : "747", "score" : 467 } { "value" : "302", "score" : 449 } ... ```
其中 value 是用户 id,score 是发帖数。可以看到 uid 为 2 的用户以 4121 个帖子位居榜首。
下面是使用 nodebb 数据库模块获取论坛发帖最多用户的完整 javascript 函数:
``` const db = require('../database'); const user = require('../user') async function getTopPosters(start, stop) { const uids = await db.getSortedSetRevRange('users:postcount', start, stop); return await user.getUsers(uids); } ```
db 类提供了一整套操作 hash、set、sorted set 和 list 的函数,源码见这里。无论你用 mongodb 还是 redis,接口都完全一致,具体说明参见这篇文章。
NodeBB 中有大量 sorted set 用于展示数据。比如话题的帖子存在 `tid:
用户相关的 sorted set:
``` key: users:joindate score: 用户注册时间戳 value: 用户 uid
key: users:online score: 用户最后在线时间戳 value: 用户 uid
key: users:postcount score: 用户发帖数 value: 用户 uid
key: users:reputation score: 用户声望值 value: 用户 uid
key: users:notvalidated score: 用户被加入此集合的时间戳 value: 用户 uid
key: users:flags score: 用户被举报次数 value: 用户 uid
key: users:banned score: 用户被封禁的时间戳 value: 用户 uid
key: users:banned:expire score: 用户封禁到期的时间戳 value: 用户 uid
key: uid:
key: username:uid score: 用户 uid value: 用户名
key: userslug:uid score: uid value: 用户的 userslug
key: email:uid score: 用户 uid value: 用户的小写邮箱
key: email:sorted
score: 0
value: <小写邮箱>:
key: username:sorted
score: 0
value: <小写用户名>:
key: user:
key: user:
key: uid:
key: cid:
分类相关的 sorted set:
``` key: categories:cid score: 分类排序 value: 分类 id
key: cid:
key: categories:name
score: 0
value: <小写分类名>:
key: cid:
key: cid:
key: cid:
key: cid:
key: cid:
话题相关的 sorted set:
``` key: topics:tid score: 话题时间戳 value: 话题 id
key: topics:posts score: 话题帖数 value: 话题 id
key: topics:votes score: 话题投票数 value: 话题 id
key: topics:views score: 话题浏览量 value: 话题 id
key: tid:
帖子相关的 sorted set:
``` key: posts:pid score: 帖子时间戳 value: 帖子 id
key: tid:
key: tid:
聊天相关的 sorted set:
```
key: uid:
key: uid:
key: chat:room:
key: chat:room:
key: chat:room:
key: chat:rooms:public score: 聊天室创建时间戳 value: 聊天室 id
key: chat:rooms:public:order score: 聊天室排序 value: 聊天室 id
key: chat:rooms score: 房间创建时间戳 value: 聊天室 id ```