Kubernetes Storage Options💣
Use this data to assist in your CSI decision. However, when using a cloud provider we suggest you use their Kubernetes CSI.
Feature Matrix💣
| Product | BB Compatible | License Type | In Ironbank | RWX/RWM Support | Airgap Compatible | Cloud Agnostic |
|---|---|---|---|---|---|---|
| Amazon EBS CSI | X | Apache License 2.0 | X | AWS Dependent | No | |
| Azure Disk CSI | Not Tested | Apache License 2.0 | X | Azure Dependent | No | |
| Longhorn v1.1.0 | X | Apache License 2.0 | X | X - Docs | Yes, uses host storage | |
| OpenEBS (jiva) | X | Apache License 2.0 | X Alpha | Manual Work Required | Yes, uses host storage | |
| Rook-Ceph | X | Rook - Apache License 2.0. Ceph - dual licensed under the LGPL version 2.1 or 3.0 | X | Manual Work Required | Yes, uses host storage | |
| Portworx | X | Tiered License - See website | X | X - Docs | Yes, uses host storage |
Benchmark Results💣
Benchmarks were tested on AWS with GP2 ebs volumes using using FIO, see example
| Product | Random Read/Write IOPS | Average Latency (usec) | Sequential Read/Write | Mixed Random Read/Write IOPS |
|---|---|---|---|---|
| Amazon EBS CSI | 2997/2996. BW: 128MiB/s / 128MiB/s | 1331.61 | 129MiB/s / 131MiB/s | 7203/2390 |
| Azure Disk CSI | ||||
| Longhorn v1.1.0 | 6155/1551 BW: 230MiB/s / 96.3MiB/s | 1042.53 | 319MiB/s / 130MiB/s | 3804/1267 |
| OpenEBS (jiva) | 2183/770. BW: 76.8MiB/s / 45.8MiB/s | 2059.55 | 132MiB/s / 98.2MiB/s | 1590/533 |
| Rook-Ceph | 10.7k/3205. BW: 503MiB/s / 148MiB/s | 548.36/s | 496MiB/s / 154MiB/s | 6664/2228 |
| Portworx 2.6 | 3016/19.3k. BW: 74.5MiB/s / 85.1MiB/s | 1337.31 | 113MiB/s / 124MiB/s | 35.1k/11.1k |
Amazon EBS CSI💣
Requirements💣
- Must be using AWS
Notes💣
- Apache License 2.0
- Very easy to install and use, apply CSI spec and you are ready.
Azure Disk CSI💣
Requirements💣
- Must be using Azure
Notes💣
- Apache License 2.0
- Very easy to install and use, apply CSI spec and you are ready.
Longhorn💣
Requirements💣
- RWX requires
nfs-commonto be installed on the nodes. Longhorn RWX Docs
Notes💣
- Apache License 2.0
- Easiest to install
- Built-in backup tool.
- Documented airgap install process. Docs
- GUI provides data and observability; replica status, cluster health status, backup status, and backup initiation/recovery.
- Native backup to S3 or NFS
OpenEBS💣
Requirements💣
- Blank and un-partitioned attached disk(s)
- RWX is in Alpha and requires work. OpenEBS RWX Docs
Notes💣
- Very flexible, supports multiple storage designs.
| Application requirements | Storage Type | OpenEBS Volumes |
|---|---|---|
| Low Latency, High Availability, Synchronous replication, Snapshots, Clones, Thin provisioning | SSDs/Cloud Volumes | OpenEBS Mayastor |
| High Availability, Synchronous replication, Snapshots, Clones, Thin provisioning | Disks/SSDs/Cloud Volumes | OpenEBS cStor |
| High Availability, Synchronous replication, Thin provisioning | hostpath or external mounted storage | OpenEBS Jiva |
| Low latency, Local PV | hostpath or external mounted storage | Dynamic Local PV - Hostpath |
| Low latency, Local PV | Disks/SSDs/Cloud Volumes | Dynamic Local PV - Device |
| Low latency, Local PV, Snapshots, Clones | Disks/SSDs/Cloud Volumes | OpenEBS Dynamic Local PV - ZFS |
Rook-Ceph💣
Requirements💣
- Blank and un-partitioned attached disk(s)
Notes💣
- Rook - Apache License 2.0.
- Ceph - dual licensed under the LGPL version 2.1 or 3.0
- Very Fast
Portworx💣
Requirements💣
- Blank and un-partitioned attached disk(s)
Notes💣
- Portworx Essentials is free up to 5nodes, 5TB Storage, 500 volumes
- Portworx Enterprise and PX-Backup require paid licenses
- Best Mixed IOPS, average read/write performance
- Install is very picky about the container runtime hostpath
- Tested on Konvoy 1.6.1 due to Portworx issues when using RKE2
Last update:
2022-08-05 by Ryan Thompson