{
    "componentChunkName": "component---node-modules-lekoarts-gatsby-theme-minimal-blog-core-src-templates-post-query-tsx",
    "path": "/learning-blue-store-a-gentle-introduction-to-cephs-storage-engine",
    "result": {"data":{"post":{"slug":"/learning-blue-store-a-gentle-introduction-to-cephs-storage-engine","title":"Learning BlueStore: A Gentle Introduction to Ceph's Storage Engine","date":"24.08.2026","tags":[{"name":"ceph","slug":"ceph"},{"name":"bluestore","slug":"bluestore"},{"name":"storage","slug":"storage"},{"name":"distributed-systems","slug":"distributed-systems"}],"description":null,"canonicalUrl":null,"body":"var _excluded = [\"components\"];\nfunction _extends() { return _extends = Object.assign ? Object.assign.bind() : function (n) { for (var e = 1; e < arguments.length; e++) { var t = arguments[e]; for (var r in t) ({}).hasOwnProperty.call(t, r) && (n[r] = t[r]); } return n; }, _extends.apply(null, arguments); }\nfunction _objectWithoutProperties(e, t) { if (null == e) return {}; var o, r, i = _objectWithoutPropertiesLoose(e, t); if (Object.getOwnPropertySymbols) { var s = Object.getOwnPropertySymbols(e); for (r = 0; r < s.length; r++) o = s[r], t.includes(o) || {}.propertyIsEnumerable.call(e, o) && (i[o] = e[o]); } return i; }\nfunction _objectWithoutPropertiesLoose(r, e) { if (null == r) return {}; var t = {}; for (var n in r) if ({}.hasOwnProperty.call(r, n)) { if (e.includes(n)) continue; t[n] = r[n]; } return t; }\n/* @jsxRuntime classic */\n/* @jsx mdx */\n\nvar _frontmatter = {\n  \"title\": \"Learning BlueStore: A Gentle Introduction to Ceph's Storage Engine\",\n  \"date\": \"2026-08-24T00:00:00.000Z\",\n  \"draft\": false,\n  \"tags\": [\"ceph\", \"bluestore\", \"storage\", \"distributed-systems\"]\n};\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n    props = _objectWithoutProperties(_ref, _excluded);\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, \"Hey folks, it's been some time since I last blogged. I have decided to blog more\\nfrequently now (and I hope to keep this conviction up and not let it become yet\\nanother post of me \\\"trying to blog frequently\\\" ).\"), mdx(\"p\", null, \"Recap: I have been working at Ceph for almost 2 years now and recently have\\nstarted working with the BlueStore team. The last few months since the switch,\\nit feels like a haze has lifted from me. I am increasingly getting interested\\nand awed by the working of Storage Systems and the principles that back it. I\\nthink I finally found an area that actually speaks to me, or is it too early to\\nmake this proclamation (?) - only future will tell.\"), mdx(\"p\", null, \"Paraphrasing the general consensus\\n(\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://skeptics.stackexchange.com/questions/8742/did-einstein-say-if-you-cant-explain-it-simply-you-dont-understand-it-well-en\"\n  }, \"1\"), \",\\n\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://addyosmani.com/blog/write-learn/\"\n  }, \"2\"), \") - \", mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"\\\"To Write is to Learn\\\"\"), \", thus\\nhere we are - I will be writing blogs about my work with BlueStore, Storage\\nsystems and whatever learnings I keep having on the way.\"), mdx(\"p\", null, \"Disclaimer, I am pretty new to the field of (Distributed) Storage Systems so I\\nam bound to make many mistakes, if you find me doing so - please feel free to\\nlet me know. I accept any/all criticisms.\"), mdx(\"p\", null, \"The first blog would be about a small Intro to BlueStore.\"), mdx(\"p\", null, \"Let's Begin....\"), mdx(\"hr\", null), mdx(\"h2\", null, \"BlueStore\"), mdx(\"p\", null, \"Ceph stores data on physical machines using OSDs (Object Storage Daemons). These\\ndaemons need a backend storage engine - something that takes the request \\\"write\\nthis object\\\" and actually puts bytes into a disk. BlueStore is that engine. Ceph\\nhas 3 other storage engines before BlueStore, but notably - BlueStore replaced\\nFileStore backend and became the default in Ceph Luminous. Why did BlueStore\\nbecome the default, that is a post for another day.\"), mdx(\"p\", null, \"The key architectural decision: \", mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"BlueStore stores object data directly on raw\\nblock devices rather than a conventional file system\"), \". BlueStore manages\\nallocation and I/O itself, while RocksDB stores metadata to describe that data.\"), mdx(\"h2\", null, \"How BlueStore fits in the Ceph Storage Stack\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"Client (librados / RBD / CephFS / RGW)\\n         \\u2502\\n    RADOS (distributed object layer)\\n         \\u2502\\n    OSD daemon (one per disk)\\n         \\u2502\\n    ObjectStore API  \\u2190 BlueStore implements this\\n         \\u2502\\n    BlueStore\\n    \\u251C\\u2500\\u2500 RocksDB (metadata: onodes, extent maps, freelist)\\n    \\u2502     \\u2514\\u2500\\u2500 BlueFS (minimal FS for RocksDB)\\n    \\u251C\\u2500\\u2500 Allocator (BitmapAllocator / FreelistManager)\\n    \\u2502\\n    \\u2514\\u2500\\u2500 Block Device(s) (raw disk for object data)\\n          \\u251C\\u2500\\u2500 BlueFS region (RocksDB WAL + DB)\\n          \\u2514\\u2500\\u2500 Data region (object data, raw I/O)\\n\")), mdx(\"p\", null, \"The \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"ObjectStore\"), \" is the storage backend interface that defines how the OSD\\nshould interact with its local storage backend. This is the contract that\\nBlueStore implements. This contract has operations like (\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"read/write\"), \",\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"setattr/getattr\"), \", \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"transactions\"), \" etc). This interface allows Ceph to swap\\ndifferent backends when needed.\"), mdx(\"h2\", null, \"Components of BlueStore\"), mdx(\"h3\", null, \"RocksDB\"), mdx(\"p\", null, \"RocksDB is a high performance embedded key-value store. This is used as the\\nmetadata store for BlueStore. BlueStore uses RocksDB because it is fast and\\ncrash-consistent, to store metadata like:\"), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"onodes\"), \": per-object metadata (size, extent map, checksums)\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"extent maps\"), \": which block ranges in disk belong to which objects\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"freelist\"), \": which blocks are allocated/free\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"deferred write data\"), \": small writes staged temporarily before writeback.\")), mdx(\"p\", null, \"When a client writes 1 MB object called \\\"foo\\\", BlueStore needs to update several\\npieces of metadata atomically:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"The \", mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"onode\"), \" for \\\"foo\\\" (object size, checksum)\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"The \", mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"extent map\"), \" (object bytes 0 -  1 MB -> disk blocks 2801 - 3056)\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"The \", mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"freelist\"), \" (mark blocks 2801-3056 as used)\")), mdx(\"p\", null, \"These are all separate key-value entries in RocksDB. RocksDB provides\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"WriteBatch\"), \" API that lets one group multiple key-value operations into a\\nsingle atomic unit. Either all three keys get written or none of them do.\\nRocksDB guarantees this by writing the entire batch to its WAL\\n(write-ahead-log). If the process crashes mid-way, on recovery RocksDB replays\\nthe WAL and either the whole batch is there or it isn't.\"), mdx(\"p\", null, \"This way, if any crash happens between any of the steps, BlueStore can reclaim it!\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"TL;DR: Its job is to store everything BlueStore needs to find and describe\\nthe data.\")), mdx(\"h3\", null, \"Allocators\"), mdx(\"p\", null, \"BlueStore manages raw block devices directly, so it needs its own mechanism to\\ntrack which blocks are free and which are occupied. BlueStore has two related\\npieces involved in allocation:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"a \", mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"persistent representation\"), \" of allocation state, and\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"an \", mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"in-memory allocator\"), \" used to make fast allocation decisions.\")), mdx(\"p\", null, \"This is a two-tier design:\"), mdx(\"h4\", null, \"FreelistManager - the persistent record\"), mdx(\"p\", null, \"This is the on-disk truth about what's free. It lives inside RocksDB as\\nkey-value pairs. (Note the \\\"freelist\\\" in the previous section) When\\nBlueStore allocates or frees block, the freelist manager update goes into the\\nsame \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"WriteBatch\"), \" as the rest of the metadata - this makes the allocation\\nchanges atomic with the write itself.\"), mdx(\"p\", null, \"We can't query the freelist manager (RocksDB key) on every write as it's not\\nperformant. That's where the in-memory allocator comes in.\"), mdx(\"p\", null, \"Note: There's also another path called as Null FreelistManager, which I aim to\\nunderstand and cover in a future blog post.\"), mdx(\"h4\", null, \"In-memory Allocator - the fast decision maker\"), mdx(\"p\", null, \"On mount, BlueStore reads the entire freelist from RocksDB and builds an\\nin-memory structure for fast allocations. This is what actually answers the\\nquestion \\\"give me 64 KB of contiguous free space\\\".\"), mdx(\"p\", null, \"BlueStore has various implementations available for in-memory allocators:\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"AVLAllocator\"), \" (two AVL trees), \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"BitmapAllocator\"), \" (hierarchical bitmap) and\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"HybridAllocator\"), \" (AVL tree with bitmap overflow). Each of these allocators make\\ndifferent tradeoffs between speed, memory use and allocation quality.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"\\u250C\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2510\\n\\u2502              TIER 1: Allocator  (lives in RAM)              \\u2502\\n\\u2502                                                             \\u2502\\n\\u2502  Fast in-memory data structure                              \\u2502\\n\\u2502  Answers: \\\"give me 64KB of free space\\\" instantly            \\u2502\\n\\u2502  Lost on every restart \\u2014 rebuilt from Tier 2 on mount       \\u2502\\n\\u2514\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2518\\n                          \\u25B2\\n                          \\u2502 rebuilt on mount\\n                          \\u2502\\n\\u250C\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2510\\n\\u2502         TIER 2: BitmapFreelistManager  (lives in RocksDB)   \\u2502\\n\\u2502                                                             \\u2502\\n\\u2502  Persistent bitmap \\u2014 one bit per disk block                 \\u2502\\n\\u2502  Source of truth \\u2014 survives crashes and restarts            \\u2502\\n\\u2502  Updated atomically inside every RocksDB transaction        \\u2502\\n\\u2514\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2518\\n\")), mdx(\"h3\", null, \"BlueFS\"), mdx(\"p\", null, \"RocksDB is designed to work with POSIX files - it creates, appends, reads and\\ndeletes SST files and WAL logs. BlueStore has no OS filesystem and its whole\\npoint was to avoid general purpose filesystem, so the options were to:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Put RocksDB on XFS/ext4 - defeats the purpose, reintroduces the filesystem\\noverhead\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Rewrite RocksDB to work with raw block I/O - requires massive effort, hard to\\nmaintain upstream\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Build a minimal filesystem that does only what RocksDB needs\")), mdx(\"p\", null, \"Option 3 was the approach taken, and BlueFS was created for this purpose. BlueFS\\nis a minimal, purpose-built filesystem-like layer that exists for one reason: to\\ngive RocksDB a filesystem like interface on top of raw block device.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"Block Device\\n\\u251C\\u2500\\u2500 BlueFS region\\n\\u2502     \\u251C\\u2500\\u2500 RocksDB WAL files\\n\\u2502     \\u251C\\u2500\\u2500 RocksDB SST files\\n\\u2502     \\u2514\\u2500\\u2500 BlueFS journal\\n\\u2502     (BlueFS manages allocation here)\\n\\u2502\\n\\u2514\\u2500\\u2500 BlueStore data region\\n      \\u251C\\u2500\\u2500 Object data\\n      (BlueStore's Allocator manages allocation here)\\n\")), mdx(\"p\", null, \"BlueFS allows multi device optimizations. BlueStore can be configured with three\\ntypes of block devices\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"WAL device\"), \": RocksDB WAL - Latency sensitive - fast NVMe ideal\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"DB device\"), \": RocksDB SST files - Read-heavy, moderate size - SSD works well\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"Main device\"), \": Object data - Bulk capacity - HDD is also fine\")), mdx(\"p\", null, \"All three things are by default on the same device, but BlueFS allows users to\\nsplit them up if needed. For eg: You can put the WAL and DB on NVMe while the\\nactual Object data lives on the HDD. BlueFS manages the WAL and DB device\\nregions and BlueStore manages the main device directly. If the DB device fills\\nup, BlueFS will allocate from the main (slow) device.\"), mdx(\"p\", null, \"On default setting, where all data share the same physical device - BlueStore\\nand BlueFS manages their own regions independently. BlueStore knows where\\nBlueFS's region starts and ends and doesn't touch it. There's also a\\ncoordination mechanism as mentioned previously: If BlueFS needs more space, it\\ncan request from BlueStore's free pool.\"), mdx(\"hr\", null), mdx(\"h2\", null, \"Conclusion\"), mdx(\"p\", null, \"I focused this blog on trying to give a very brief introduction as to what\\nBlueStore is. In future blogs - I aim to dive deeper into various other aspects,\\nand keep sharing what I learn from my daily workings!\"), mdx(\"p\", null, \"Ciao until then!\"));\n}\n;\nMDXContent.isMDXComponent = true;","excerpt":"Hey folks, it's been some time since I last blogged. I have decided to blog more\nfrequently now (and I hope to keep this conviction up and…","timeToRead":4,"banner":null}},"pageContext":{"slug":"/learning-blue-store-a-gentle-introduction-to-cephs-storage-engine","formatString":"DD.MM.YYYY"}},
    "staticQueryHashes": ["2744905544","3090400250","318001574"]}