Ceph – Introduction

Ceph is an impressive and at times daunting storage product. If you are reading this though, chances are good that you are looking to learn more about it and trying to determine if and why it could be right for your application or use case. Throughout this article, we will dive into the origins of Ceph, examine its functionality and the underlying technology, and go over some potential deployment scenarios. This is the first in a series of articles focused on open source storage technologies, such as Ceph, that aims to bring a level of understanding to software defined storage products in the open source realm, leveraging our extensive experience with open source software and technologies.


    Ceph is an open source distributed storage solution offering tremendous flexibility and adaptability. Originally conceived as a doctoral thesis by Sage Weil, responsibility for the software passed to RedHat in 2014 when it acquired Weil’s consulting organization that sprung up around supporting it. The name Ceph, derived from “cephalopod”, came about due to the software’s similarity in functionality to the bilateral functions of Octopus and Squid tentacles.
    Since the first release, Argonaut on July 3rd, 2012, Ceph has seen several developmental iterations as new technologies are incorporated. With deployment tools now available for Openstack and Proxmox, iterations that support presenting iSCSI storage directly to virtualization platforms, as well as a capable API, Ceph deployments can be found at some of the world’s largest organizations – powering hyperscale block and file storage. 

    Ceph is growing in popularity and attracting significant interest from many big enterprises because it is currently the only storage solution that provides all of the following capabilities:

    Above: An Example of Ceph cluster, scaling out as you grow

    • SOFTWARE-DEFINED – Software-defined storage is a policy-based provisioning and management of data storage independent of the underlying hardware. Software-defined storage typically includes a form of storage virtualization to separate the storage hardware from the software that manages it. With Ceph the environment may also provide policy management for features such as data deduplication, replication, thin provisioning, snapshots and backup.
    • ENTERPRISE-CLASS – Ceph is designed to meet the demands of a large organizations for availability, compatibility, reliability, scalability, performance and security, among other things. It supports both scale up and scale out options making it highly flexible and nearly limitless in its scalability potential.
    • UNIFIED STORAGE – Ceph provides Block + Object + File storage which provides greater flexibility (most other storage products are block only, file only, object only, or file+block; the trifecta offered by CEPH is very rare).
    • OPEN-SOURCED – Open source enables technology agility, typically offering multiple ways to solve problems. So, instead of waiting for a vendor to deliver a missing capability within the framework of the software, you can create it yourself. Open source is also generally much more cost effective and can more easily allow organizations to start smaller and scale. Much of the ideology behind open-source solutions breeds a collaborative and involved professional community that is highly responsive and mutually supportive. Not to mention that Open source is the direction of the future. Web, mobile, and cloud solutions are increasingly built on open source infrastructure.

    Why Ceph?

    Initial work on clustered filesystems was begun in the late 1990’s and early 2000’s. One of the first implementations to leverage scalable file systems was a product called Lustre. Over the years, several other Lustre spin-offs came out, including the likes of GlusterFS, GPFS, XtreemFS, and OrangeFS just to name a few. These filesystems all suffered from a focus on implementing POSIX compliant mount-points for filesystems and lacked major integrative API’s – at least initially.
    Ceph was conceived without the notion that POSIX compliant filesystems were required – a benefit of coming of age in the cloud-era. Leveraging RADOS, Ceph could scale block devices that weren’t held back by metadata constraints. This vastly improved performance but left out those looking for a way to mount a large filesystem based on Ceph. That was until the release of Ceph Jewel (version 10.2.0), where CephFS was considered Stable and complete – allowing the deployment of POSIX mountable filesystems.

    Ceph is capable of block, object, and file storage and it is scale-out, meaning multiple Ceph storage nodes (servers) cooperate to present a single storage system that easily handles many petabytes (1PB = 1,000 TB = 1,000,000 GB). It is also able to increase both performance and capacity at the same time, leveraging the underlying commodity hardware components that serve as it’s foundation.
    Ceph has many basic enterprise storage features including replication, erasure coding, snapshots, thin provisioning, tiering (ability to shift data between flash and hard drives – i.e. caching), and self-healing capabilities. In order to do that, Ceph leverages several components that we’ll delve into below.
    As of Ceph Nautilus (v14.2.0) there are now five primary daemons, or services, that integrate to provide the base set of Ceph services. These are:

    1. ceph-mon. The Monitor provides for exactly what the name implies – monitoring cluster health. The monitor also tells OSD’s where to place data during replication and holds the master CRUSH map – more on that later.
    2. ceph-osd. The OSD is the underlying data storage platform for Ceph, leveraging the XFS filesystem and a physical disk to store block data presented to it from clients.
    3. ceph-mds. The MDS daemon provides for converting Ceph block data into a POSIX compliant mount point for storage files as you would a traditional filesystem.
    4. ceph-mgr. The MGR daemon presents monitoring and management information about the status of the cluster. 
    5. ceph-rgw. The RGW daemon is an HTTP API daemon that presents the underlying clusters as buckets of storage in a way that it remains compatible with Amazon’s S3 or Openstack’s Swift API tools.

    From a terminology standpoint, there are a few important ones to know with Ceph;

    Ceph is built around the CRUSH Algorithm  and supports multiple access methods (file, block, object). The CRUSH algorithm determines where objects go on what OSD’s, and can be reversed to pull those same blocks out for access requests.

    Ceph leverages Reliable, Autonomic, Distributed, Object Store (or RADOS) comprised of self-healing, self-managing storage nodes. OSD Daemons discussed earlier form part of the RADOS cluster.
    A placement group (PG) is a CRUSH map determined identifier for where data should go in the cluster. The number of PG’s in a cluster determine it’s size, and several calculators exist for right-sizing your cluster. PG’s can be set by the administrator or can be auto scaled depending on cluster usage.
    RBD stands for RADOS block device and can be used within Ceph to create image blocks for virtualization like KVM and Xen. By leveraging librados, a RADOS compatible API, VM access is facilitated not by iSCSI or NFS, aging IP based tech, but rather through a storage API.


    Potential Deployment Scenarios

    Ceph is an extraordinarily flexible and coherent storage solution, as we’ve already described. Access to Ceph-stored objects can be accomplished in multiple ways, and as a result Ceph has a surprising number of use-cases which its predecessors generally lack. 

    Ceph can be deployed as a drop-in S3/Swift object storage replacement. Through it’s RADOS Gateway, objects stored in Ceph can be accessed using http GET requests as well as the plethora of Amazon S3 API toolkits available. 

    Ceph can also be leveraged in virtualization environments, including VMWare and other proprietary virtualization platforms, either via the librados API or via iSCSI/NFS. 

    CephFS can be deployed to provide POSIX compliant mountpoints for operating systems that need access to large scale file systems.

    All of this leads to organizational consolidation, as a single storage platform can be leveraged to suit a wide variety of computing and storage needs.




    Throughout this series, we will dig deeper into the technology that powers Ceph, and expand on the specific use cases and why they are so unique.  Ceph is an extremely powerful technology that continues to improve with each iteration, and we’re excited to help you leverage it in your business.

    Learn more about our consulting services. Our experts are available to speak with you and are always ready to assist.


    Ceph Storage Specialists Are Ready To Help

    Contact our team of infrastructure and cloud hosting experts to discuss your business needs at any stage in your decision making process. From planning, to post implementation we have experts that can help you in any capacity.