s3fs-fuse

Unofficial notes taken during my integration of the tool with a Clowder extractor , road map I plan to use:

Install on Mac

Run


brew install --cask macfuse
brew install gromgit/fuse/s3fs-mac

Turns out running on mac comes with some overhead security issues and stuff, moving to my vm

Install on Linux


sudo apt install s3fs

I then proceeded to connect to the minio bucket and was able to see all the files in the directory

Docker-compose setup

To the existent Clowder setup, I added an ubuntu docker image, the image needed to be added with privilege to access the FUSE system since it needs requires special privileges and kernel access.


ubuntu-client:
    image: ubuntu:22.04
    networks:
      - clowder2
    restart: unless-stopped
    command: tail -f /dev/null
    environment:
      MINIO_ENDPOINT: minio-nginx:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ./scripts:/scripts
    # Added privileged mode and device mappings for FUSE support
    privileged: true
    devices:
      - /dev/fuse:/dev/fuse
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    depends_on:
      - minio-nginx

Get into the ubuntu terminal


docker-compose exec ubuntu-client bash

Here I ran the following commands that gave me access to minio filesystem


apt-get update && apt-get install -y fuse
apt-get update && apt-get install -y curl wget # optional but to check connection
apt install s3fs
echo minioadmin:minioadmin > .miniocred #username password
chmod 600 .miniocred
mkdir minio-mount # folder that will be mount point
s3fs clowder ./minio-mount -o passwd_file=/.miniocred,use_path_request_style,url=http://minio-nginx:9000/

Security Question - Since I am mounting minio file system, does this not mean the extractor has access to files beyond the user’s allowed ones?

Dockerfile and entrypoint script


FROM ubuntu:22.04

# Install required packages
RUN apt-get update && apt-get install -y \
    fuse \
    s3fs \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Create mount directory
RUN mkdir /clowderfs

# Create entrypoint script
# The script mounts the bucket using credentials passed as environment variables.
# It ensures that the MinIO container is ready before the mount.

COPY entrypoint.sh /entrypoint.sh

# Make the script executable
RUN chmod +x /entrypoint.sh

# Set the entrypoint
ENTRYPOINT ["/entrypoint.sh"]


#!/bin/bash
set -e

# Ensure required environment variables are set
if [[ -z "${MINIO_ACCESS_KEY}" || -z "${MINIO_SECRET_KEY}" || -z "${MINIO_ENDPOINT}" ]]; then
    echo "ERROR: MINIO_ACCESS_KEY, MINIO_SECRET_KEY, and MINIO_ENDPOINT must be set."
    exit 1
fi

# Create credentials file using environment variables
echo "${MINIO_ACCESS_KEY}:${MINIO_SECRET_KEY}" > /.miniocred
chmod 600 /.miniocred

# Wait for the MinIO endpoint to become available
until curl -s "http://${MINIO_ENDPOINT}" >/dev/null; do
    echo "Waiting for MinIO at ${MINIO_ENDPOINT}..."
    sleep 3
done

# Mount the S3 bucket
s3fs clowder /clowderfs \
    -o passwd_file=/.miniocred \
    -o use_path_request_style \
    -o url=http://${MINIO_ENDPOINT}/ \
    -o allow_other

# Keep the container running
exec tail -f /dev/null