Enterprise, Large Scale File Services

~ Matt Smith

Admittedly, the world of file services has changed since Novell ruled the roost with Netware. All sorts of new buzzwords exist: Web Content Management, Enterprise Document Management, Document Archiving, Knowledge Management … but you still can’t beat simple file storage service like Windows offers natively. Except, it is really hard to provide that kind of service at a very large scale.
So here is the question — how do I provide a file service with the following requirements:

  • Must scale beyond 25,000 users (potentially 100,000), each with private “home” directories, plus whatever permutations of group space can be imagined.
  • Must support large amounts of storage, including individual files of several hundred gigabytes, user/group quotas of several terabytes.
  • Must support access from OSX, Windows, and Linux such that applications on these systems can natively open, read, write files — in other words, similar to simple CIFS access, though a non-native client to support this functionality is acceptable.
  • Must support some level of access from mobile devices, including Android, iPhone/iPad, Windows Mobile, and ideally Blackberry too.
  • Must provide a rich “sexy-looking” web interface.
  • Must provide consistent abstract interface — in other words, scaling across hundreds of servers is acceptable, as long as users never need to be told “connect to server #17 for X, and server #53 for Y”.  There should be some sort of abstracted virtual filesystem.
  • Must support user-controllable ACLs to facilitate sharing and security.
  • Must be accessible by non-technical end users with very little handholding – should be “intuitive”.
  • Must allow integration with backup solution that can provide file-level restoration.
  • Should allow for storage of data to be accessed by Linux and Windows servers, such as user generated web content, HPC-generated research data, etc.
  • Should allow for attachment of metadata for searching.
  • Should allow integration with backup solution that allows end-user to perform file-level restoration.

Some have tried to convince me that Windows DFS can do all this, but I have yet to see a deployment that actually encompasses all of the above.  Anyone have any references?

I am quite intrigued by OpenAFS, using the filedrawers web interface, and possibly using the Samba gateway to avoid deploying the OpenAFS client to every machine — anyone with any experience doing this?  Anyone serve OpenAFS data out over DAV via Apache, mod_dav, and mod_waklog?  Is filedrawers or DAV an acceptable mobile device access mechanism?  Pitfalls?

What else should I be considering?

Leave a comment