Building and Running FiberFS

Home

Github

git clone https://github.com/fiberfs/fiberfs.git

FiberFS on Github

Building

make
FiberFS has 2 optional but recommended package dependencies: openssl and zlib. Don't forget to add "-j4" (4 being the number of available cores to use) to all your make commands.

Testing

make test

Config

At minimum, your FiberFS config file (fiberfs.conf) just needs to define your S3 endpoint: 

S3_HOST = my-bucket.s3.region123.cloud-provider.com
S3_REGION = region123
S3_ACCESS_KEY = ACCESS_KEY_STRING
S3_SECRET_KEY = SECRET_KEY_STRING
More configuration parameters are documented below.

Running

./fiberfs [fiberfs.conf] [mount_point]

Logging

./fiberfs_log [mount_point]

S3 Configuration

S3_HOST - Hostname or IP of the S3 endpoint.
S3_REGION - S3 region.
S3_ACCESS_KEY - S3 access key.
S3_SECRET_KEY - S3 secret key.
S3_PREFIX - S3 prefix to use on all operations. Optional. Must start with a leading "/", ex: "/bucket/fs".
S3_TLS - Use TLS for S3 communication. Defaults to true.
S3_PORT - S3 port. Defaults to 443 if TLS is enabled, otherwise 80.

FS Configuration

FS_ROOT_MODE - Access mode for the mount directory in octal. Defaults to 755.
FS_ROOT_UID - UID for the mount directory. Defaults to the UID of the FiberFS process.
FS_ROOT_UID - GID for the mount directory. Defaults to the GID of the FiberFS process.
DIRECTORY_TTL_SEC - How long to cache directories before looking for new updates. Defaults to 5 seconds.
MAX_DIRECTORIES - Maximum number of directories to keep in memory at anytime. If a directory is not available in memory, it will be loaded from cache or S3. Defaults to 250. A future feature will introduce a memory based limit (ex: 100MB).
LOG_SIZE - Size of the in-memory log in bytes. Defaults to 131072 (128KB).
ASYNC_TASKS - Maximum number of burstable concurrent tasks. After this number is reached, FiberFS will degrade to fair share across all operations. Defaults to 16. Note that this value is shared across S3, cache, and cluster opertions.
FUSE_ALLOW_OTHER - Allow all users to see the mount. Access is still enforced via FS_ROOT_MODE. Defaults to false, mounts can only be seen by user running FiberFS process. If mounting as non-root, user_allow_other must be enabled in fuse.conf to use this setting.
FUSE_WRITEBACK_CACHE - Enable writeback caching in the kernel page cache. Defaults to false. Keep this parameter disabled if you require globally atomic O_APPEND behavior.

Cache Configuration

CACHE_ROOT - Cache location. Defaults to /tmp/fiberfs_cache. The actual cache root is a hashed subdirectory under this location.
CACHE_SIZE_MB - Cache size in MB. Defaults to 25 (MB).
CACHE_CHUNK_WRITE_THRU - Cache chunks and files during write operations. Defaults to true. If you have a write heavy workload which you are not planning on reading locally, disable this parameter to bypass local cache writes and save IO.

Cluster Configuration

CSTORE_SERVER - Set to true to enable the cluster server (cache store). Defaults to false. Cstore logging is placed in a memory log mounted on the cache root.
CSTORE_SERVER_ADDRESS - Network address to listen on.
CSTORE_SERVER_PORT - Network port to listen on.
CSTORE_SERVER_TLS - Enable and only accept TLS for all cluster communication. Note that enabling TLS will disable the ability to splice data off the network which will have a negative performance impact on the cluster. Defaults to false.
TLS_CERT_FILE - TLS certificate file (full chain) in PEM format. Required if CSTORE_SERVER_TLS is enabled.
TLS_PRIVATE_KEY_FILE - TLS private key file in PEM format. Required if CSTORE_SERVER_TLS is enabled.
CLUSTER - A comma seperated list of ADDRESS:PORT pairs of all the cluster members.
CLUSTER_TLS - Use TLS for cluster communication. Defaults to false.

Note that the cluster server does not need to be enabled on all FiberFS clients, the cluster can be broken up into servers and satellites. Lets say you have 4 hosts with a mounted FiberFS that you want to cluster. Their IPs are 10.0.0.1 thru 10.0.0.4 and each server has 50GB of cache space. Your fiberfs.conf cache and cluster section will look like this: 

CACHE_ROOT = /var/cache/fiberfs
CACHE_SIZE_MB = 50000

CSTORE_SERVER = true
CSTORE_SERVER_ADDRESS = 10.0.0.X
CSTORE_SERVER_PORT = 14667

CLUSTER = 10.0.0.1:14667, 10.0.0.2:14667, 10.0.0.3:14667, 10.0.0.4:14667
Replace 10.0.0.X with the actual IP of the host. This cluster will now have 200GB of cache and store copies of all roots, indexes, chunks, and full files. This is now a fully functioning 4 host cluster. Now lets say we have other hosts which we want to participate in the cluster, but we don't want them to act as servers. These are called satellites. Their fiberfs.conf only needs the single cluster configuration line and no server config (default cache config works well here): 
CLUSTER = 10.0.0.1:14667, 10.0.0.2:14667, 10.0.0.3:14667, 10.0.0.4:14667
These satellites will still have a default 25MB cache in /tmp/fiberfs_cache that will locally cache roots and indexes only. A copy of all roots, indexes, chunks and full files will be stored on the cluster and is available to all servers and satellites. This has all the benefits of a full cluster with a more compact cache footprint.

CDN Configuration

CDN_ENDPOINT - Hostname or IP endpoint for the CDN. If no port is specified, port 443 is used if TLS is enabled, otherwise 80. Multiple endpoints can be specified using a comma seperated list in which case FiberFS will automatically shard traffic across them.
CDN_TLS - Use TLS for CDN communication. Using TLS prevents spliced IO operations. Defaults to true.
CDN_ALLOW_PUT - Send PUT requests to the CDN. By default this is false and FiberFS will send PUT requests directly to S3. Enabling this allows the CDN to perform write thru caching. A PUT request can be stored and permanently cached at any point during the transaction, even before seeing the response code.

The CDN can cache all requests for an indefinite amount of time using just the URL as the cache key. This includes PUT requests, if configured. If any error is encountered, simply return a 500+ HTTP error and FiberFS will automatically retry the request directly to S3. Note that FiberFS does not send all its traffic to the CDN, synchronization sensitive traffic is always routed directly to S3.

Content

Home