Recently encountered a very interesting problem with Azure file share. Would like to document the findings here.
Conclusion
Don’t migrate to different technologies without thorough testing. For example, if your application is NFS-based, don’t change it to CIFS/SMB-based without thorough testing, because features, performance may be very different.
Background
We are migrating a legacy client application into Azure. The application is very old and is linux and NFS-based (all OS, python dependencies, code are stored in NFS server). There is no clear-cut dependency for each component. Therefore, we end up with 60GB binaries in unzipped formats. Also, the application uses lots of symlinks as versioning strategy. We know it is not good, but unwinding dependencies would take much longer and requires application team support. Application team prefers lift-and-shift approach to run VMs in Azure, instead re-architecting and using modern managed service.
Considerations
- Client’s solution does not allow us to directly access Azure cli or portal, so we cannot save VM image that contains 60GB binaries. We have to build VM image onprem with client API (using QEMU) and ship to Azure. Apparently QEMU does not like building large amounts of unzipped binaries with lots of symlinks and constantly failed. In the end, either the VM image can only contain zipped binaries or we use mounted approach in Azure.
- At this time, NFS protocol of Azure file share is still in preview and can only be provisioned by command line (not even from portal), and there is no terraform support yet. We need to use client’s terraform-based solution, so we decide to try CIFS first
Test result and analysis
RHEL8 CIFS improved upon RHEL7 a lot, but still much worse than onprem NFS performance, while RHEL8 NFS is onpar with onprem NFS performance.
Finding
Since the application uses lots of symlinks, in order to keep same behavior on Azure, we used the following option as described in article below.
,mfsymlinksBut according to our test, although the application still can run, the performance is much worse.
At this point, NFS protocol of Azure file share is still in preview and has limitations described in article below (e.g. no terraform support, no Data plane REST APIs support, no Storage explorer, AzCopy support, etc.)
Slow copy performance on Linux clients
When we mount Azure file share on a cloud VM, and test copy speed, we found only 0.5Gb/s, much worse than the promised throughput. After research, it seems that we are impacted by single-threaded issue.
Microsoft actually recommends using azcopy to interact with Azure file share through RESTful data plane. If you still want to use plain Linux command, for premium file shares, enable SMB Multichannel and use multi-threaded application.
