Exporting HDFS via FUSE and Samba: Experiment and Performance Results (Updated)
Some years ago I experimented with exposing HDFS to non-Hadoop clients. After playing with libhdfs and NFS and not getting the results I wanted, the next idea was: why not try Samba? Samba 3.x was stable, and most operating systems could mount a CIFS share without additional software.
The main goal was to understand the setup and performance characteristics of this scenario, because Samba has many tuning options. The environment used in this experiment was RHEL 5.7 with the stock Samba packages.
1. Environment and packages
# rpm -qa | grep samba
samba-3.0.33-3.29.el5_7.4.x86_64
samba-common-3.0.33-3.29.el5_7.4.x86_64
2. Mount HDFS via FUSE
As with the NFS export in section 8 below, HDFS is first mounted locally using hadoop-fuse-dfs. The mount point used here was /123/hdfs, configured in /etc/fstab:
# cat /etc/fstab
[...]
hadoop-fuse-dfs#dfs://NAMENODE:9000 /123/hdfs fuse usetrash,rw 0 0
Check the mount:
# mount
[...]
fuse on /123/hdfs type fuse (rw,nosuid,nodev,allow_other,default_permissions)
And list the directory:
# ls -la /123
total 16
drwxr-xr-x 3 root root 4096 Dec 9 16:36 .
drwxr-xr-x 27 root root 4096 Dec 9 12:11 ..
drwxr-xr-x 5 hdfs nobody 4096 Dec 9 02:14 hdfs
At this point HDFS is available on the local node under /123/hdfs via FUSE.
3. Samba configuration
The next step is to configure Samba to export this FUSE-mounted HDFS path to remote clients.
# cat /etc/samba/smb.conf
[global]
bind interfaces only = yes
deadtime = 15
default case = lower
disable netbios = yes
interfaces = eth0
dns proxy = no
workgroup = HDFS
server string = Samba Server Version %v
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
load printers = no
max connections = 30
strict sync = no
sync always = no
syslog = 1
syslog only = yes
security = user
smb passwd file = /etc/samba/smbpasswd
[hdfs]
comment = HDFS
path = /123/hdfs
public = yes
writable = yes
printable = no
create mask = 0744
force user = hdfs
force group = nobody
For authentication, I created a Samba user and password. In this setup I reused the HDFS system user (hdfs, group nobody):
# smbpasswd -a username
Finally, restart the Samba service:
# service smb restart
4. Mounting the share from a client
On another RHEL 5.7 server, the Samba share is mounted to /test using CIFS:
# mount -t cifs -o username=hdfs,rw //SAMBASERVER/hdfs /test
Password: HERE_THE_PASSWORD
Verify the mount:
# ls -la /test/
total 8
drwxr-xr-x 5 hdfs nobody 0 Dec 9 02:14 .
drwxr-xr-x 25 root root 4096 Dec 9 15:03 ..
drwxr-xr-x 3 hdfs nobody 0 Dec 9 02:12 mapred
drwxr-xr-x 3 hdfs nobody 0 Dec 9 02:13 opt
drwxr-xr-x 6 hdfs nobody 0 Dec 9 15:56 user
Now the HDFS filesystem from the test cluster is exported via Samba and accessible to CIFS clients. So far, so good.
5. Read performance test with rsync
The first performance test focused on read throughput. I used rsync to copy a collection of smaller log files from HDFS (via Samba) to the local filesystem:
# cd /tmp/rsync-test
# rsync -av /test/hdfs/user/flume/weblogs/2011-12-07/ .
sent 20478888644 bytes received 92606 bytes 17377158.46 bytes/sec
total size is 20475835998
(19GB, ~16 MB/s)
Number of files synced:
# find . -type f | wc -l
4665
The result shows that reading many small files via Samba on top of HDFS FUSE achieved roughly 16 MB/s on this setup.
6. Write performance test with dd and cp
For write performance, I created a large test file:
# dd if=/dev/zero of=/tmp/13GB bs=128M count=100
Then copied it into the CIFS-mounted share, measuring the time:
# time cp /tmp/13GB /test/hdfs/user/
real 7m57.864s
user 0m0.328s
sys 0m20.602s
This corresponds to roughly 27 MB/s write throughput via Samba on top of FUSE-mounted HDFS.
Check permissions and ownership in HDFS:
# hadoop dfs -ls /user
Found 1 item
-rw-r--r-- 3 hdfs supergroup 13421772800 2011-12-09 15:56 /user/13GB
7. Comparison with direct SCP
To compare, I used scp to write the same file directly to the node where HDFS was mounted:
# scp /tmp/13GB hdfs@SAMBASERVER:/123/hdfs/user
13GB 100% 13GB 47.8MB/s 04:28
Direct SCP to the FUSE mount was significantly faster (~47.8 MB/s) than writing through Samba (~27 MB/s). The additional overhead from Samba clearly costs performance.
8. The Alternative: Exporting the FUSE Mount over NFS
Before Samba, I tried NFS. The idea is the same: mount HDFS locally with FUSE and libhdfs, then export that mount. Here is the full NFS setup, based on the CDH3 packages of that time. Replace namenode.local and <PORT> with your own values.
8.1 Install FUSE and libhdfs and create a mountpoint
yum install hadoop-0.20-fuse.x86_64 hadoop-0.20-libhdfs.x86_64
mkdir /hdfs-mount
8.2 Test the FUSE mount
hadoop-fuse-dfs dfs://namenode.local:<PORT> /hdfs-mount -d
If the mount works, the output looks like this:
INFO fuse_options.c:162 Adding FUSE arg /hdfs-mount
INFO fuse_options.c:110 Ignoring option -d
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.10
flags=0x0000000b
max_readahead=0x00020000
INFO fuse_init.c:101 Mounting namenode.local:<PORT>
INIT: 7.8
flags=0x00000001
max_readahead=0x00020000
max_write=0x00020000
unique: 1, error: 0 (Success), outsize: 40
Once you see Success, stop the foreground process with Ctrl+C. This command is only for testing.
8.3 Mount at boot time
echo "hadoop-fuse-dfs#dfs://namenode.local:<PORT> /hdfs-mount fuse usetrash,rw 0 0" >> /etc/fstab
Then test it:
# mount -a
# mount
[...]
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
fuse on /hdfs-mount type fuse (rw,nosuid,nodev,allow_other,default_permissions)
Each FUSE process runs a JVM. You can tune its heap and other runtime settings in:
/etc/default/hadoop-0.20-fuse
8.4 Export over NFS (trusted networks only)
This export is not secure. User IDs and permissions are mapped at the OS level, so only use it in trusted environments. First check the UID and GID of the user you want to map to:
# id hdfs
uid=104(hdfs) gid=105(hdfs) groups=105(hdfs),104(hadoop) context=root:staff_r:staff_t:SystemLow-SystemHigh
Then export only the HDFS user directory in /etc/exports:
# cat /etc/exports
/hdfs-mount/user (fsid=111,rw,wdelay,anonuid=104,anongid=105,sync,insecure,no_subtree_check,no_root_squash)
fsid=111: unique filesystem ID (seeman 5 exports)anonuid=104,anongid=105: map anonymous users to thehdfsuser and groupinsecure: allow connections from non-privileged portsno_subtree_check,no_root_squash: turn off subtree checks and root squashing
Exporting only /hdfs-mount/user keeps paths like /mapred safe from accidental changes. Restart NFS and mount it on a client:
# service nfs restart
# mount -t nfs <NFS_SERVER_HOST>:/hdfs-mount/user /mnt/hdfs-user
Every file operation goes through FUSE and libhdfs into HDFS calls, and ownership maps to the local hdfs user. Don’t work as root on the clients. In my tests this only worked reliably from Linux kernel 2.6.27 upwards.
Conclusion
This experiment shows that it is technically possible to export a FUSE-mounted HDFS filesystem via Samba and access it from CIFS clients with acceptable read/write performance. For some operational and ad-hoc tasks, this can make it easier to interact with HDFS as a (limited) cluster storage from systems that are not HDFS-aware.
However, the additional layers (FUSE + Samba, or FUSE + NFS) introduce overhead and complexity. For production workloads or high-throughput scenarios, native HDFS clients, WebHDFS, HDFS NFS gateways, object stores, or modern data access layers are usually the better choice.
For further tuning ideas, see Samba optimization guides such as: https://calomel.org/samba_optimize.html
Need help with distributed systems, backend engineering or data platforms? See my consulting services.
Founder of Scalytics, co-creator and PMC member of Apache Wayang, and author of KafScale. I write about Kafka, streaming, data platforms, IoT and AI agent infrastructure in production.