{
    "componentChunkName": "component---node-modules-lekoarts-gatsby-theme-minimal-blog-core-src-templates-post-query-tsx",
    "path": "/why-is-blue-store-the-way-it-is-part-2-cephs-backends-in-detail",
    "result": {"data":{"post":{"slug":"/why-is-blue-store-the-way-it-is-part-2-cephs-backends-in-detail","title":"Why Is BlueStore the Way It Is? Part 2: Ceph's Backends in Detail","date":"21.10.2026","tags":[{"name":"ceph","slug":"ceph"},{"name":"bluestore","slug":"bluestore"},{"name":"storage","slug":"storage"},{"name":"distributed-systems","slug":"distributed-systems"}],"description":null,"canonicalUrl":null,"body":"var _excluded = [\"components\"];\nfunction _extends() { return _extends = Object.assign ? Object.assign.bind() : function (n) { for (var e = 1; e < arguments.length; e++) { var t = arguments[e]; for (var r in t) ({}).hasOwnProperty.call(t, r) && (n[r] = t[r]); } return n; }, _extends.apply(null, arguments); }\nfunction _objectWithoutProperties(e, t) { if (null == e) return {}; var o, r, i = _objectWithoutPropertiesLoose(e, t); if (Object.getOwnPropertySymbols) { var s = Object.getOwnPropertySymbols(e); for (r = 0; r < s.length; r++) o = s[r], t.includes(o) || {}.propertyIsEnumerable.call(e, o) && (i[o] = e[o]); } return i; }\nfunction _objectWithoutPropertiesLoose(r, e) { if (null == r) return {}; var t = {}; for (var n in r) if ({}.hasOwnProperty.call(r, n)) { if (e.includes(n)) continue; t[n] = r[n]; } return t; }\n/* @jsxRuntime classic */\n/* @jsx mdx */\n\nvar _frontmatter = {\n  \"title\": \"Why Is BlueStore the Way It Is? Part 2: Ceph's Backends in Detail\",\n  \"slug\": \"/why-is-blue-store-the-way-it-is-part-2-cephs-backends-in-detail\",\n  \"date\": \"2026-10-21T00:00:00.000Z\",\n  \"draft\": false,\n  \"tags\": [\"ceph\", \"bluestore\", \"storage\", \"distributed-systems\"]\n};\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n    props = _objectWithoutProperties(_ref, _excluded);\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, \"Hey folks, in the\\n\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"/why-is-blue-store-the-way-it-is-part-1-why-file-systems-fall-short\"\n  }, \"last blog\"), \"\\nwe saw that there are three main problems with using file systems as a storage\\nbackend:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Inefficient transactions\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Slow metadata operations\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Less support for new storage hardware\")), mdx(\"p\", null, \"Until now we have looked at the problems in the abstract. In this post we will\\nwalk through Ceph's backends in order and see which of these problems they had\\nand how they got around them.\"), mdx(\"p\", null, \"As a reminder, this series is split into three parts:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, mdx(\"a\", {\n    parentName: \"strong\",\n    \"href\": \"/why-is-blue-store-the-way-it-is-part-1-why-file-systems-fall-short\"\n  }, \"Part 1\"), \":\"), \"\\nWhat a distributed system needs from its storage backend, and why file\\nsystems struggle to provide it.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"Part 2 (this post):\"), \" Ceph's backends in order (EBOFS \\u2192 FileStore on Btrfs\\n\\u2192 FileStore on XFS \\u2192 NewStore) and how each of them ran into these problems.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"Part 3:\"), \" How all of these learnings shaped BlueStore, and the price it\\npays for owning the disk.\")), mdx(\"p\", null, \"While reading the paper, one pattern that showed up was: \\\"each workaround fixed\\none problem and created a new one\\\". This says a lot about how software evolves\\nand how there are always drawbacks with any solution, we just need to know which\\ndrawbacks we are comfortable with.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"2005 - EBOFS (custom)\\n2008 - FileStore/Btrfs (borrow FS transactions)\\n2012 - FileStore/XFS (user-space WAL, directory split)\\n2015 - NewStore (RocksDB as WAL)\\n2017 - BlueStore (raw disk + RocksDB)\\n\")), mdx(\"h2\", null, \"EBOFS\"), mdx(\"p\", null, \"The first backend on Ceph was EBOFS (Extent and B-Tree-based Object File\\nSystem). This was a custom user-space object store that ran directly on the raw\\ndisk.\\n(\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://github.com/ceph/ceph/commit/18d9132a053846d7fd24ec512a83d4da106d6ed7\"\n  }, \"commit\"), \"\\nthat added this store). This was mostly a small research system and got\\ndeprecated in 2008 in favour of using Btrfs.\"), mdx(\"p\", null, \"I was not able to find many resources on the specific issues EBOFS faced - Ceph\\nhad just started at that time, so there might not have been very many documented\\nproblems. What the paper does say is that EBOFS lacked features like\\ntransactions, deduplication, checksums and transparent compression, which Btrfs\\nalready had.\"), mdx(\"h2\", null, \"FileStore on Btrfs\"), mdx(\"p\", null, \"Btrfs came with features like transactions, checksums and copy-on-write that\\nwere required for a production-ready storage backend. Instead of implementing\\nall of these features by itself, Ceph decided to borrow them.\"), mdx(\"p\", null, \"EBOFS hence got replaced by FileStore, which is an ObjectStore implementation\\nthat stores objects as files on a local file system. The FileStore's mapping is\\nbelow:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \" Ceph                      File system\\n \\u2500\\u2500\\u2500\\u2500                      \\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\n collection (PG)    \\u2500\\u2500\\u25BA    directory\\n object             \\u2500\\u2500\\u25BA    file\\n object attributes  \\u2500\\u2500\\u25BA    xattrs (later LevelDB, when xattrs got too big)\\n\")), mdx(\"p\", null, \"As we discussed in the previous post, it is a must for the storage backend to\\nhave transactions. Let us see how FileStore went about implementing them.\"), mdx(\"h3\", null, \"Transactions: first attempt - borrow Btrfs's\"), mdx(\"p\", null, \"Btrfs exposed its internal transactions through a pair of ioctls,\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"BTRFS_IOC_TRANS_START\"), \" and \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"BTRFS_IOC_TRANS_END\"), \". The first FileStore wrapped\\neach Ceph transaction between them:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"ioctl(TRANS_START)\\n    write(obj_A, ...)          \\u2190 kernel applies this now\\n    setxattr(obj_A, ...)       \\u2190 and this\\n    write(pglog, ...)          \\u2190 OSD process dies here, or hits ENOSPC\\n    write(pginfo, ...)\\nioctl(TRANS_END)\\n\")), mdx(\"p\", null, \"As we saw in \", mdx(\"em\", {\n    parentName: \"p\"\n  }, \"Leveraging the File System's Internal Transactions\"), \" in Part 1,\\nthis approach breaks down the moment the OSD process dies (a software crash or a\\nKILL signal) in the middle of a transaction while the kernel keeps running.\\nBtrfs has no rollback, so it commits the half-done operations, leaving the data\\nin an inconsistent state.\"), mdx(\"p\", null, \"One fix, \", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://lwn.net/Articles/361439/\"\n  }, \"proposed by Sage Weil in 2009\"), \", was\\nto introduce a single system call specifying the entire transaction. This way\\nthe kernel does the validation up front and can reserve the space and metadata\\nfor the worst case before modifying anything. It also provides a clean\\nall-or-nothing boundary.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"ioctl(TRANSACTION, [\\n  {WRITE,    obj_A, off, data},\\n  {SETXATTR, obj_A, \\\"_\\\", info},\\n  {WRITE,    pglog, ...},\\n  {WRITE,    pginfo, ...},\\n])\\n\")), mdx(\"p\", null, \"This proved too costly for the following reasons:\"), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Computing the worst-case scenario reservations for a mixed list of operations\\nis hard. This also over-reserves and hurts everyone sharing the file system.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Large transactions would hold a Btrfs transaction open longer, which delays\\nevery other process.\")), mdx(\"p\", null, \"It was never merged, and Btrfs later deprecated the transaction ioctls\\naltogether.\"), mdx(\"h3\", null, \"Transactions: second attempt - Btrfs snapshots\"), mdx(\"p\", null, \"The second attempt at fixing the rollback issue was to use the snapshot feature\\nof Btrfs. The Btrfs snapshot is a cheap, point-in-time copy of a subvolume.\"), mdx(\"p\", null, \"Btrfs stores everything in copy-on-write B-trees. A subvolume is basically a\\npointer to a tree root.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"subvol \\\"current\\\"  \\u2500\\u2500\\u25BA  root R1\\n                        \\u251C\\u2500\\u2500 A\\n                        \\u2514\\u2500\\u2500 B \\u2500\\u2500 C (obj_A data)\\n\")), mdx(\"p\", null, \"Taking a snapshot doesn't copy any data. It creates a second root pointer to the\\nsame tree and bumps the reference counts:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"\\\"current\\\"  \\u2500\\u2500\\u25BA  R1 \\u2500\\u2510\\n\\\"snap_100\\\" \\u2500\\u2500\\u25BA  R1' \\u2534\\u2500\\u2500\\u25BA A, B, C   (shared, refcount = 2)\\n\")), mdx(\"p\", null, \"When you write to \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"obj_A\"), \", copy-on-write kicks in - a new separate block called\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"C'\"), \" gets created with the data. Then all the parent blocks above C are also\\ncreated as new blocks (in this example \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"B'\"), \"), including the root itself (\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"R2\"), \"),\\nand \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"current\"), \" now points to the new root. The shared data remains as is.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"\\\"current\\\"  \\u2500\\u2500\\u25BA  R2 \\u2500\\u2500\\u252C\\u2500\\u2500 B' \\u2500\\u2500 C'   (new data)\\n                     \\u2502\\n                     \\u2514\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2510\\n                                \\u25BC\\n                                A    (shared, one copy on disk)\\n                                \\u25B2\\n                     \\u250C\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2500\\u2518\\n                     \\u2502\\n\\\"snap_100\\\" \\u2500\\u2500\\u25BA  R1' \\u2500\\u2534\\u2500\\u2500 B \\u2500\\u2500 C     (old data, untouched)\\n\")), mdx(\"p\", null, \"As you can see here the \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"snap_100\"), \" still points to the old data.\"), mdx(\"p\", null, \"For this, FileStore added its own user-space journal (the WAL from Part 1) on\\ntop of Btrfs and used the snapshots as rollback points. The following is what\\nhappens in FileStore:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Every few seconds: snapshot the Btrfs subvolume holding the objects, and\\nrecord \", mdx(\"inlineCode\", {\n    parentName: \"li\"\n  }, \"this snapshot = journal position N\")), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"On crash:\", mdx(\"ol\", {\n    parentName: \"li\"\n  }, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"roll back to the last snapshot, disk is now exactly at position N\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"replay the journal from N, every txn is applied exactly once\")))), mdx(\"p\", null, \"Note that, when we roll back to the last snapshot - the entire disk along with\\nits data is rolled back - so replay always starts from a known state. The\\noperations are still non-idempotent, but since they are never replayed on top of\\ntheir own effects, the non-idempotent replay problem disappears.\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"No transaction is ever applied twice\"), \" in this approach.\"), mdx(\"h4\", null, \"Problems:\"), mdx(\"p\", null, \"The design was sound but Btrfs at that time wasn't ready to handle Ceph's\\nworkload. Ceph's workload hit Btrfs kernel bugs and unmountable file systems in\\nproduction clusters (ref:\\n\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://lore.kernel.org/all/1994850.3pprtJuiTQ@pc10/T/\"\n  }, \"1\"), \",\\n\", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"https://github.com/ceph/ceph/commit/efc235895d862ef664188583bb77d0a3f8db5baa\"\n  }, \"2\"), \").\\nConstant small overwrites and snapshots fragmented both metadata and data badly\\nand the performance degraded over time.\"), mdx(\"p\", null, \"During Ceph's years on Btrfs, the ObjectStore interface had grown so much that\\ngoing back to EBOFS wasn't practical. Instead, FileStore was ported to run on\\nXFS, ext4 and later ZFS. Of these, XFS became the de facto backend since it\\nscaled better and had faster metadata performance.\"), mdx(\"h2\", null, \"FileStore on XFS\"), mdx(\"p\", null, \"XFS was more stable than Btrfs, but it had no cheap way to create snapshots. And\\nas we saw in the previous section - without snapshots, the replays will make the\\nstate of data inconsistent.\"), mdx(\"h3\", null, \"Transactions: a pure user-space WAL\"), mdx(\"p\", null, \"Since there was no snapshot, the other way to do safe transactions with replay\\ncapability was by implementing a write-ahead log in user space. This is the WAL\\nthat we described in \\\"Implementing the WAL in User Space\\\". With this approach,\\nFileStore ran into all the problems we listed previously:\"), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Double writes, every byte went into the journal and into XFS.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Slow read-modify-write, a transaction's effect was readable only after it had\\nbeen committed to the journal and applied to XFS.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Non-idempotent replay, the replay ran on whatever was already on the disk.\")), mdx(\"p\", null, \"Since the last one was more damaging, as it deals with the correctness of data,\\nCeph tried to fix that first. It did the following to fix it:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"inlineCode\", {\n    parentName: \"li\"\n  }, \"syncfs()\"), \", FileStore has to periodically flush the file system to the\\nphysical device, so that the journal can be trimmed and replay starts from a\\nknown sync point (the risky window from Part 1). On XFS, the only option was\\n\", mdx(\"inlineCode\", {\n    parentName: \"li\"\n  }, \"sync\"), \", which flushes every file system on every drive in the node - very\\nexpensive, since an OSD node usually has many drives. Hence Sage Weil added a\\nnew \", mdx(\"a\", {\n    parentName: \"li\",\n    \"href\": \"https://lwn.net/Articles/433384/\"\n  }, mdx(\"inlineCode\", {\n    parentName: \"a\"\n  }, \"syncfs()\")), \" system call to Linux that\\nsyncs just one file system.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Guards, sequence numbers were stored with objects so that replay could skip\\noperations that were already applied. This was simple for writes, but very\\nhard for multi-object operations like clone and rename. For example:\")), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"op 41: write obj_A \\\"hello\\\"\\nop 42: append obj_A \\\" world\\\"\\nop 43: write obj_B \\\"foo\\\"\\n\")), mdx(\"p\", null, \"Ops 41 and 42 were applied before the crash, and obj_A's guard says 42. Op 43\\nnever reached the disk and obj_B's guard says 38. Without the guard, op 42 would\\nrun again and obj_A would have become \\\"hello world world\\\".\"), mdx(\"p\", null, \"During Replay:\"), mdx(\"table\", null, mdx(\"thead\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"thead\"\n  }, mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Op\"), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Object's guard\"), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Decision\"))), mdx(\"tbody\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"41 write obj_A\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"42\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"41 \\u2264 42, already applied, \", mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"skip\"))), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"42 append obj_A\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"42\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"42 \\u2264 42, already applied, \", mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"skip\"))), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"43 write obj_B\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"38\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"43 > 38, not applied yet, \", mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"apply\"))))), mdx(\"p\", null, \"This works for simple cases like the one above, but verifying guards for complex\\noperations like clone and rename was hard, and the paper notes that the\\nFileStore code ended up fragile and hard to maintain.\"), mdx(\"h3\", null, \"Metadata: Splitting directories\"), mdx(\"p\", null, \"The enumeration problem was described in the \\\"Metadata Operations\\\" section of\\nPart 1. To avoid it, FileStore did not put all of a PG's objects in one\\ndirectory. The objects were rather nested in subdirectories named after the last\\nhex digits of their hash:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \" /var/lib/ceph/osd/ceph-0/current/2.3f_head/\\n \\u2514\\u2500\\u2500 DIR_F/                                   hash ends in ...F\\n     \\u2514\\u2500\\u2500 DIR_3/                               hash ends in ...3F\\n         \\u251C\\u2500\\u2500 DIR_0/                           hash ends in ...03F\\n         \\u2502   \\u2514\\u2500\\u2500 rbd\\\\udata.abc...09__head_0C11B03F__2\\n         \\u251C\\u2500\\u2500 DIR_1/                           hash ends in ...13F\\n         \\u2502   \\u2514\\u2500\\u2500 rbd\\\\udata.def...01__head_7A40913F__2\\n         \\u251C\\u2500\\u2500 DIR_2/                           hash ends in ...23F\\n         \\u2502   \\u2514\\u2500\\u2500 rbd\\\\udata.ghi...02__head_51C0D23F__2\\n         \\u2514\\u2500\\u2500 DIR_A/                           hash ends in ...A3F\\n             \\u2514\\u2500\\u2500 rbd\\\\udata.abc...05__head_E5326A3F__2\\n\")), mdx(\"p\", null, \"Listing objects in hash order now meant we just had to walk the \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"DIR_X\"), \"\\ndirectories in order and only sort a few hundred objects.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \" walk DIR_0 \\u2192 DIR_1 \\u2192 DIR_2 \\u2192 DIR_A \\u2026     (directory names are already in order)\\n      \\u2502\\n      \\u25BC\\n readdir() each SMALL leaf  \\u2192  sort a few hundred names\\n      \\u2502\\n      \\u25BC\\n objects in hash order, one small batch at a time\\n\")), mdx(\"h4\", null, \"Problem\"), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Splitting issues, when a leaf directory filled up - it had to split one level\\ndeeper, physically moving all of its files into new subdirectories. At scale\\nthis is expensive, and the entries scatter, hence the operation needs more\\nseeks.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Long names, names longer than XFS's 255-byte limit were truncated - with the\\nreal name stored in an xattr. So listing needed an extra lookup per object.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"xattr limits, object attributes outgrew what xattrs could hold.\")), mdx(\"h2\", null, \"NewStore: RocksDB as WAL\"), mdx(\"p\", null, \"In the previous sections, we saw that most of the problems occur because we are\\ntrying to pack the metadata into the file system. If we could segregate the data\\nand the metadata of objects into different stores, that could eliminate a lot of\\nproblems.\"), mdx(\"p\", null, \"Hence NewStore tried to tackle the root cause of the FileStore's problem: \", mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"the\\nfile system was being used as the source of truth for metadata\"), \".\"), mdx(\"p\", null, \"The design of NewStore is as below:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \" NewStore\\n  \\u251C\\u2500\\u2500 metadata (object info, attrs, PG log, collections)  \\u2192 RocksDB\\n  \\u2514\\u2500\\u2500 object data                                      \\u2192 files on XFS\\n\")), mdx(\"p\", null, \"With metadata being stored in a durable and atomic KV store, accessing it in a\\nsafe way becomes much easier.\"), mdx(\"h3\", null, \"Transaction: one atomic RocksDB batch\"), mdx(\"p\", null, \"FileStore had no real transactions; it faked them with a journal. NewStore used\\nRocksDB's transactions and put all the metadata changes into a single\\ntransaction.\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"1. write(F7, data); fsync(F7)    \\u2190 object data goes into a file on XFS\\n2. batch.put(\\\"O/A\\\",      {version: 42, size: 12288, data_file: F7})\\n   batch.put(\\\"pglog/42\\\", \\\"modify A v42\\\")\\n3. db.write(batch, sync=true)    \\u2190 all-or-nothing\\n\")), mdx(\"p\", null, \"If the crash happened before step 3, the data may exist on the disk but no\\nmetadata references it, and the stray data will just get picked up for cleaning\\nlater.\"), mdx(\"h3\", null, \"Read-modify-write: no waiting for the commit\"), mdx(\"p\", null, \"In FileStore, the next transaction couldn't see the previous one's result until\\nit had been fsynced to the journal and applied to XFS. But in NewStore, the new\\nstate goes directly into RocksDB. This helps us to read the state immediately\\ninstead of waiting on the previous transaction's fsync.\"), mdx(\"h3\", null, \"Non-idempotent replay: resolve the \\\"read\\\" up front\"), mdx(\"p\", null, \"FileStore's journal stored operations like \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"clone a\\u2192b\"), \", which read \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"a\"), \" at replay\\ntime. If \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"a\"), \" had already changed, replay copied the wrong data. NewStore\\nresolves that read when the transaction is prepared, not when it's replayed.\"), mdx(\"p\", null, \"For eg:\"), mdx(\"p\", null, \"If we have \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"clone a->b\"), \", if \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"a\"), \" is small - the bytes are copied into the\\ntransaction itself. The transaction then reads something like\\n\", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"b = 4 (exact value)\"), \".\"), mdx(\"p\", null, \"If \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"a\"), \" is large, no copy is done. We point \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"a\"), \" and \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"b\"), \" both to the same data and\\nmark it as read-only (copy-on-write). Any new writes then go to a new location.\\nThe transaction here only records pointers.\"), mdx(\"p\", null, \"Either way, nothing in the transaction depends on the current state of the disk,\\nso replay is always safe.\"), mdx(\"h3\", null, \"Enumeration: sorted keys instead of directories\"), mdx(\"p\", null, \"In FileStore, we stored objects under PG directories. In RocksDB, the keys are\\nalways sorted.\"), mdx(\"p\", null, \"A simplified view of the keys:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \"O/2/0C11B03F/rbd_data.abc...09\\nO/2/51C0D23F/rbd_data.ghi...02\\nO/2/7A40913F/rbd_data.def...01\\nO/2/E5326A3F/rbd_data.abc...05\\n\")), mdx(\"p\", null, \"In NewStore, every object's metadata is a key in RocksDB, and the key starts\\nwith the pool and the object's hash followed by the object name. Since RocksDB\\nalways keeps its keys sorted, all of the PG's objects sit next to each other.\"), mdx(\"h3\", null, \"Directory splitting: nothing to split\"), mdx(\"p\", null, \"Since there are no directories, there are no splitting storms. When Ceph splits\\na PG into two, NewStore doesn't move any files. The new PG is just a different\\nkey range.\"), mdx(\"h3\", null, \"Problems\"), mdx(\"p\", null, \"As explained in Part 1, NewStore had the problem of a journal on top of a\\njournal. Object data still lived in files on XFS, and RocksDB's own files lived\\non XFS too. Every \", mdx(\"inlineCode\", {\n    parentName: \"p\"\n  }, \"fsync\"), \" looked like this:\"), mdx(\"pre\", null, mdx(\"code\", {\n    parentName: \"pre\"\n  }, \" create one object in NewStore:\\n   fsync(data file)        \\u2192 flush data      + flush XFS journal   = 2\\n   fsync(RocksDB WAL)      \\u2192 flush log entry + flush XFS journal   = 2\\n                                                                   \\u2500\\u2500\\u2500\\u2500\\u2500\\n                                                     total flushes:  4\\n\\n same work on a raw disk:                                          = 2\\n\")), mdx(\"h2\", null, \"Summary of Ceph Backends\"), mdx(\"h3\", null, \"Putting it together\"), mdx(\"table\", null, mdx(\"thead\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"thead\"\n  }, mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"th\"\n  }, \"EBOFS\")), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"th\"\n  }, \"FileStore / Btrfs\")), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"th\"\n  }, \"FileStore / XFS\")), mdx(\"th\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"th\"\n  }, \"NewStore\")))), mdx(\"tbody\", {\n    parentName: \"table\"\n  }, mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Era\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"2005 \\u2013 2008\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"~2008 \\u2013 2012\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"~2012 \\u2013 2017\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"~2015 (never in production)\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Runs on\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Raw disk (user space)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Btrfs\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"XFS (also ext4, ZFS)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"XFS (data files + RocksDB files)\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"How data is stored\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Custom extents on disk\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Object = file, PG = directory\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Object = file, PG = nested hash directories\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Object data = files on XFS\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"How metadata is stored\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Custom B-trees (object table, free list)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Directories + xattrs, large attrs in LevelDB\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Directories + xattrs, large attrs and omap in LevelDB\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"All in RocksDB as sorted keys\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"How transactions work\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Its own code, owns the whole disk\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"1st: Btrfs \", mdx(\"inlineCode\", {\n    parentName: \"td\"\n  }, \"TRANS_START/END\"), \" ioctls (no rollback). 2nd: user-space journal + Btrfs snapshots, rolling back before replay\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"User-space WAL with full data journaling, \", mdx(\"inlineCode\", {\n    parentName: \"td\"\n  }, \"syncfs()\"), \" to trim, sequence-number guards for replay\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"One atomic RocksDB write batch per transaction\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Double writes\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"No\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Yes (journal + Btrfs)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Yes, every byte written twice\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Gone for new objects, still there for overwrites\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Read-modify-write\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Waits for journal commit + apply\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Waits for journal fsync + apply to XFS\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Read new state from RocksDB immediately\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Crash replay\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Safe: roll back to snapshot, replay once\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Risky: replays on top of applied state; guards are fragile\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Safe: reads resolved at prepare time, puts are idempotent\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Listing objects (enumeration)\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Own B-tree\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"inlineCode\", {\n    parentName: \"td\"\n  }, \"readdir()\"), \" on large directories: slow, unordered\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Walk hash directories, sort small leaves\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Range scan over sorted keys\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Clones / snapshots\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Cheap (Btrfs copy-on-write)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Expensive: full physical copy\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Copy-on-write (pointers + read-only data)\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"Erasure-coded overwrites\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Impractical (each OSD copies its chunk)\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"\\u2014\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"What finally broke it\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Lacked checksums, compression, CoW\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Btrfs instability, data and metadata fragmentation\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Halved bandwidth, fragile replay, splitting storms (7 min SSD / 120 min HDD stalls), slow clones, writeback latency spikes\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Journal on journal: 4 flushes per object vs 2 on raw disk (70\\u201380% slower object creation)\")), mdx(\"tr\", {\n    parentName: \"tbody\"\n  }, mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, mdx(\"strong\", {\n    parentName: \"td\"\n  }, \"What it passed on\")), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Idea of owning the disk\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"The journal, and the cost of depending on one FS's features\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"The diagnosis: the FS shouldn't be the source of truth for metadata\"), mdx(\"td\", {\n    parentName: \"tr\",\n    \"align\": null\n  }, \"Metadata in RocksDB, the core of BlueStore\")))), mdx(\"h2\", null, \"Wrapping Up\"), mdx(\"p\", null, \"Taking a look back at all of these backends, the pattern from the start of this\\npost occurs repeatedly, where each workaround fixed one problem and created a\\nnew one:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"EBOFS lacked features like transactions and checksums, so Ceph borrowed them\\nfrom Btrfs.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"Btrfs's transactions had no rollback. Snapshots fixed that, but Btrfs itself\\nwas not stable enough for Ceph's workload.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"XFS was stable, but without snapshots we needed a user-space journal, guards\\nfor replay and directory splitting for enumeration - each with its own cost.\"), mdx(\"li\", {\n    parentName: \"ol\"\n  }, \"NewStore moved the metadata into RocksDB and fixed most of FileStore's\\nproblems, but the object data and RocksDB still lived on XFS, and we ended up\\nwith a journal on top of a journal.\")), mdx(\"p\", null, \"One important take-away is, with every step - Ceph needed a little less from the\\nfile system. Maybe that is what led the developers to ask the question \\\"What if\\nwe remove the file system completely?\\\"\"), mdx(\"p\", null, \"And that is exactly what BlueStore does. In Part 3 - we will look at how all of\\nthese learnings shaped BlueStore!\"), mdx(\"p\", null, \"Ciao until then!\"));\n}\n;\nMDXContent.isMDXComponent = true;","excerpt":"Hey folks, in the\n last blog \nwe saw that there are three main problems with using file systems as a storage\nbackend: Inefficient…","timeToRead":8,"banner":null}},"pageContext":{"slug":"/why-is-blue-store-the-way-it-is-part-2-cephs-backends-in-detail","formatString":"DD.MM.YYYY"}},
    "staticQueryHashes": ["2744905544","3090400250","318001574"]}