OpenStack Trove supports database backups to object storage as it is a suitable medium.  These database backups, however, can create files that are undesirably large and can eat up a system’s disk space if you are not careful.

To avoid running out of disk space on the underlying virtual machines the data is streamed directly using the Swift client during the backup process.  Tesora DBaaS Platform (which is based on OpenStack Trove) also supports Ceph as an alternate object storage backend.

I was recently working with a customer who had a problem when uploading files to Ceph storage via its Swift-compatible API. A PUT request with no content length value was being rejected with a “411 – Content Length required”. This shouldn’t be happening as the Swift API should accept such requests as they are necessary for operations involving large files. Needless to say, we were concerned that there was a possible incompatibility between Swift’s API and Ceph’s implementation of the Swift API.

To reproduce the issue, I used the following python script:

import requests

url = "http://<endpoint>/<container>/<file>"

headers = {"X-Auth-Token": "<token>"}

data = ["str1", "str2"]

res = requests.request("PUT", url, headers=headers, data=iter(data))

print("{}: {}".format(res.status_code, res.reason))

For which we would expect the output:

201: Created

But instead, we got

411: Length Required

So what went wrong?

It turns out that the user didn’t install Ceph’s custom version of FastCGI as per their guidelines [1]. The fix was to change the FastCGI package on the RadosGW to the Ceph custom version.

Here’s what the unsupported version looks like:

~$: dpkg -l | grep -i libapache2-mod-fastcgi

ii  libapache2-mod-fastcgi 2.4.7~0910052141-1.1 amd64 Apache 2 FastCGI module 
for long-running CGI scripts

Just run the following to upgrade FastCGI. This will automatically restart the apache2 server:

~$: echo deb http://gitbuilder.ceph.com/libapache-mod-fastcgi-deb-$
(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee
/etc/apt/sources.list.d/ceph-fastcgi.list ~$: sudo apt-get update ~$: sudo apt-get install libapache2-mod-fastcgi

The Ceph version is identifiable as follows:

~$: dpkg -l | grep -i libapache2-mod-fastcgi

ii  libapache2-mod-fastcgi 2.4.7~0910052141-ceph1 amd64 Apache 2 
FastCGI module for long-running CGI scripts

After upgrading to the Ceph FastCGI, the PUT operations ran as expected and our user was able to successfully store their backups on Ceph.

 

[1] http://docs.ceph.com/docs/hammer/install/install-ceph-gateway/#apache-fastcgi-w-100-continue