Managing Multi-Protocol File Data Access Workflows with Cross-Protocol Permissions

When it comes to multi-protocol file data access, one of the challenges that Qumulo solves centers around mixed-protocol workflows, where users access the same files over NFS3, NFSv4.1, and SMB protocols. A key challenge in situations like this is how to handle file permissions.

NFS3 users accessing their files over expect a different permissions model than Windows users accessing their files over SMB or NFSv4.1. In order to create the smoothest workflow possible for customers in multi-protocol environments, Qumulo has developed a unique Cross-Protocol Permissions (XPP) framework that enables users on Windows, Linux, and macOS platforms to collaborate safely without the use of elaborate permissions safeguards and inheritance schemes.

Qumulo’s multi-protocol permissions model addresses inherent incompatibilities between POSIX and Access Control List (ACL) permissions to provide users in heterogeneous environments with as seamless a permissions experience as possible no matter what protocol they use.

What makes this problem so hard to solve?

File-system users tend not to think too much about permissions; we just expect them to work. But having permissions “just work” requires a lot of thinking from an engineering perspective when you have two completely different protocols with distinct expectations working in the same environment.

POSIX vs ACL Permissions

ACL permissions are much finer-grained than POSIX permissions. POSIX permissions are represented through mode bits that are embedded in the metadata of each file and directory in a POSIX dataset. You’ve seen these before if you’ve ever run stat or ls -l from the command line. POSIX mode bits can express rights for reading, writing, and executing files / directories for the file owner, group owner, and everyone else.

In the POSIX world, there are three basic permission levels, and they mean different things when applied to files versus directories.

SMB and NFSv4.1 permissions are represented by Access Control Lists (ACLs). An ACL consists of one or more Access Control Entries (ACEs), each of which contains a trustee, the user / group to which it applies, and a set of rights that the trustee is allowed or denied.

So for example, an ACE might express something along the lines of DENY Read to the user account named bob.

Some important points about ACLs:

• ACLs are evaluated in order. The first ACE in an ACL to apply to a trustee wins over any subsequent ACEs that may apply to that trustee.
• The possible rights in Windows are finer-grained than just read, write, and execute— in total, there are 14 possible rights, including a “full control” right that equates to the 13 other rights combined.
• ACLs apply per file / directory, and entries can be passed down from parent directories to their children. ACEs that are passed down from parent directories are referred to as “inherited,” and ACEs that are applied directly to a file in question “explicit.” ACEs have flags that denote how and whether they are inherited.
• Windows enforces the concept of a canonical ACL that dictates the order in which explicit and inherited ACEs should appear in an ACL. A canonically-ordered ACL looks like this:

Explicit DENYs
Explicit ALLOWs
Inherited DENYs
Inherited ALLOWs

Cross Protocol Permissions on Qumulo

Given the pretty significant differences between POSIX mode bits and ACLs, how does Qumulo deal with permissions? We have our own version of the ACL, called the QACL. It maps almost 1:1 to the industry-standard ACL, and it stores and understands a superset of the rights that can be expressed in both POSIX and Windows.

Inconsistencies between the ACL and POSIX permissions models can be boiled down to four operations: displaying mode bits, changing permissions, creating files, and changing the owner of a file. In this post, we’ll specifically look at displaying mode bits and changing permissions to understand how Qumulo solves these inconsistencies.

Before going more in depth about specific permissions operations, it’s important to note that one of the underlying challenges that XPP, Qumulo’s multi-protocol model, addresses is how to determine the user or users to which a set of permissions applies. This is true regardless of protocol: in a highly available distributed file system like Qumulo’s, where there are potentially multiple sources of identity (e.g., LDAP, Active Directory, Qumulo’s local users / groups), it can be difficult to tell who’s who because of the way users are stored and identified internally.

With XPP, Qumulo introduced ID equivalence, which uses POSIX UID / GID as a common identifier across identity sources: Active Directory with POSIX extensions or user-defined mappings, OpenLDAP, and Qumulo’s local users / groups. This makes it possible to compare these values to figure out who someone is and which groups they are members of.

Displaying POSIX Mode

Now, recall that Qumulo stores permissions in the form of QACLs. This means that we don’t store POSIX mode as a value on disk. When users accessing Qumulo over NFS want to see the mode bits for a certain file, we generate them from a QACL. This is straightforward when the QACL only specifies trustees that align exactly with POSIX users / groups / everyone, but when there are other, “extra” trustees, it becomes more complicated.

In this simple ACL, it’s easy to know what POSIX mode to display. The owner, group owner, and Everyone all have read rights, which translates to 444. But what if our ACL looked like this?

How do we handle the “extra” ACE that points to alice? In order to display the most accurate POSIX mode possible, we need to know whether alice is file owner and / or belongs to the file’s group. Finding out both of these pieces of information is possible, but expensive, and would cause a big performance hit for the fairly common operation of displaying POSIX mode.

In particular, in order to find out whether alice is in the file’s group, we would need to perform recursive group membership expansion for both the file’s group and for alice in addition to querying all identity sources to discover whether alice is file owner.

Qumulo approach is to do ID equivalence checks, the less expensive of the two operations, to see if an “extra” ACE is actually extra, or if it’s the file owner, but otherwise adds the extra trustee’s permissions to the Everyone bit. So assuming that alice is not the file owner in the above ACL, upon stat-ing the file, a user would see the mode bit 447, with alice’s rights folded into the Everyone bit.

This might seem unusual at first— it’s not actually true that everyone has read, write, and execute permissions to the file. Imagine, though, that we instead showed 444 as the mode for this file. Someone looking at that mode might assume that nobody had write or execute permissions. But alice does have those permissions, and leading the user to believe otherwise introduces a security risk. By showing the most permissive mode possible for a given QACL, Qumulo lets users looking at POSIX permissions know that someone, if not everyone, has the permissions denoted in the Everyone bit.

Changing POSIX Mode

Another problem with making ACLs and POSIX permissions work together is how to handle a chmod coming over NFS. Remember, QACLs potentially have rights beyond POSIX read / write / execute, as well as extra trustees and complicated inheritance schemes.

For the file to which this QACL applies, charlie is the owner. The object inherit flag on alice’s ACE means that it should be passed down to any children files / directories, and the inherit-only flag on bob’s means that it should be passed down and that it should not affect bob’s permissions on the current file. How do we change this QACL when someone does chmod 555 on it?

With XPP we use an algorithm that looks at both the requested mode and the pre-existing QACL, and manipulates the QACL to honor the intent of the chmod while also preserving the inheritance structure of the pre-existing QACL. As an example, the resulting QACL after applying the mode 555 to the one above would be:

What happened here? The rights have been changed to reflect the requested mode, 555. Notice also that there was a DENY entry on the file owner in the previous QACL that no longer exists because it contradicted the chmod.

Now let’s look at the other ACEs. Why are there now two ACEs for alice?

Before / after

The first ACE preserves the permission denied on a non-POSIX right in the original ACE (take ownership is an ACL-only concept), and the second one is just a copy of the first one, marked inherit-only so that it can be passed onto child files and directories without affecting the current file. Because bob’s ACE was inherit-only to begin with, it doesn’t affect the permissions in this file anyway, and remains the same after the chmod.

The Bottom Line: Simplifying Multi-Protocol File Data Access for Users

Qumulo’s Cross Protocol Permissions model is a unique approach to managing multi-protocol access to common data for both POSIX and ACL-based clients, providing  an option to our users that is simple, requires no configuration, and just works.

Learn More

• Read the Technical Overview and learn about Qumulo Scale AnywhereTM file services
• Subscribe to the Qumulo blog for customer stories, technical insights, industry trends and product news.

Contact us

• Take a test drive. Demo Qumulo in our new, interactive Hands-On Labs or request a free trial.
• Subscribe to the Qumulo blog for customer stories, technical insights, industry updates and product news.