By Jacqueline Kong

One of the challenges that Qumulo tries to solve around mixed-protocol workflows where users access the same files over both NFS and SMB protocols is how to handle file permissions. POSIX users accessing their files over NFS expect a different set of permissions behavior than Windows users accessing their files over SMB. This spring, in order to create the smoothest workflow possible for customers with mixed-protocol environments, Qumulo introduced Cross Protocol Permissions.

Cross Protocol Permissions (XPP) is a permissions mode consisting of many decisions made by many engineers over the course of more than a year. It addresses inherent incompatibilities between POSIX and NT permissions to provide mixed-protocol users with as seamless a permissions experience as possible. What made this problem so hard to solve?

As users of file systems, we tend not to think too much about permissions— we just expect them to work. But having permissions “just work” requires a lot of thinking from an engineering perspective when you have two completely different protocols with distinct expectations working in the same environment.

Background Information

NT permissions are much finer-grained than POSIX permissions. POSIX permissions are represented through mode bits. You’ve seen these before if you’ve ever run stat or ls -l on the command line. POSIX mode bits can express rights for reading, writing, and executing files/directories for the file owner, group owner, and everyone else.

Read Write Execute
Files Read file Append, Modify Execute file
Directories List directory Create/rename/delete children Traverse directory (look at children)

In the POSIX world, there are three basic permissions, and they mean different things when applied to files versus directories.

On the other hand, NT permissions are represented by Access Control Lists (ACLs). An ACL consists of one or more Access Control Entries (ACEs), each of which contains a trustee, the user/group to which it applies, and a set of rights that the trustee is allowed or denied.

So for example, an ACE might express something along the lines of DENY bob read.

Some important points about ACLs:

  • ACLs are evaluated in order. The first ACE in an ACL to apply to a trustee wins over any subsequent ACEs that may apply to that trustee.
  • The possible rights in Windows are finer-grained than just read, write, and execute— in total, there are 14 possible rights, including a “full control” right that equates to the 13 other rights combined.
  • ACLs apply per file/directory, and entries can be passed down from parent directories to their children. We call ACEs that were passed down from parent directories “inherited,” and ACEs that were otherwise applied directly to a file in question “explicit.” ACEs have flags that denote how and whether they are inherited.
  • Windows enforces the concept of a canonical ACL that dictates the order in which explicit and inherited ACEs should appear in an ACL. A canonically-ordered ACL looks like this:

Explicit DENYs
Explicit ALLOWs
Inherited DENYs
Inherited ALLOWs

The possible permissions in Windows.

Qumulo’s Solution

Given the pretty drastic differences between POSIX mode bits and NT ACLs, how does Qumulo deal with permissions? We have our own version of the ACL, called the QACL. It maps almost 1:1 to the NT ACL, and it stores and understands a superset of the rights that can be expressed in POSIX and Windows.

Inconsistencies between the NT and POSIX permissions models can be boiled down to four operations: displaying mode bits, changing permissions, creating files, and changing the owner of a file. In this post, we’ll specifically look at displaying mode bits and changing permissions to understand how Cross Protocol Permissions solves these inconsistencies.

Before going more in depth about specific permissions operations, it’s important to note that one of the underlying challenges that Cross Protocol Permissions addresses is how to determine the user or users to which a set of permissions applies. This is true regardless of protocol: in an environment like Qumulo, where there are potentially multiple sources of identity (e.g., LDAP, Active Directory, Qumulo’s local users/groups), it can be difficult to tell who’s who because of the way users are stored and identified internally.

With Cross Protocol Permissions, Qumulo introduced ID equivalence, which uses POSIX UID/GID as a common identifier across identity sources: Active Directory with POSIX extensions or user-defined mappings, OpenLDAP, and Qumulo’s local users/groups. This makes it possible to compare these values to figure out who someone is and which groups they are members of.

Displaying POSIX Mode

Now, recall that Qumulo stores permissions in the form of QACLs. This means that we don’t store POSIX mode as a value on-disk. When users accessing Qumulo over NFS want to see the mode bits for a certain file, we generate them from a QACL. This is straightforward when the QACL only specifies trustees that align exactly with POSIX user/group/everyone, but when there are other, “extra” trustees, it becomes more complicated.

ALLOW      file owner                read
ALLOW      file group owner     read
ALLOW      Everyone                read

In this simple ACL, it’s easy to know what POSIX mode to display. The owner, group owner, and Everyone all have read rights, which translates to 444. But what if our ACL looked like this?

ALLOW      file owner              read
ALLOW      file group owner    read
ALLOW      Everyone              read
ALLOW     alice                       read, execute, write file

How do we handle the “extra” ACE that points to alice? In order to display the most accurate POSIX mode possible, we need to know whether alice is file owner and/or belongs to the file’s group. Finding out both of these pieces of information is possible, but expensive, and would cause a big performance hit for the fairly common operation of displaying POSIX mode.

In particular, in order to find out whether alice is in the file’s group, we would need to perform recursive group membership expansion for both the file’s group and for alice in addition to querying all identity sources to discover whether alice is file owner.

What our team decided to do instead is make a trade-off: Qumulo does ID equivalence checks, the less expensive of the two operations, to see if an “extra” ACE is actually extra, or if it’s the file owner, but otherwise adds the extra trustee’s permissions to the Everyone bit. So assuming that alice is not the file owner in the above ACL, upon stat-ing the file, a user would see the mode bit 447, with alice’s rights folded into the Everyone bit.

This might seem unusual at first— it’s not actually true that everyone has read, write, and execute permissions to the file. Imagine, though, that we instead showed 444 as the mode for this file. Someone looking at that mode might assume that nobody had write or execute permissions. But alice does have those permissions, and leading the user to believe otherwise introduces a security risk. By showing the most permissive mode possible for a given QACL, Qumulo lets users looking at permissions over NFS know that someone, if not everyone, has the permissions denoted in the Everyone bit.

Changing POSIX Mode

Another problem that the team encountered when tackling the problem of making NT and POSIX permissions work together was how to handle a chmod coming over NFS. Remember, QACLs potentially have rights beyond POSIX read/write/execute, as well as extra trustees and complicated inheritance schemes.

DENY        alice                     read, take ownership (Object inherit)
DENY        charlie                  execute/traverse
ALLOW     charlie                  read, write file
ALLOW     charlie’s group     read
ALLOW     Everyone             read contents
ALLOW     bob                      read, write (Inherit-only)

For the file to which this QACL applies, charlie is the owner. The object inherit flag on alice’s ACE means that it should be passed down to any children files/directories, and the inherit-only flag on bob’s means that it should be passed down and that it should not affect bob’s permissions on the current file. How do we change this QACL when someone does chmod 555 on it?

With XPP, the team developed an algorithm that looks at both the requested mode and the pre-existing QACL, and manipulates the QACL to honor the intent of the chmod while also preserving the inheritance structure of the pre-existing QACL. As an example, the resulting QACL after applying the mode 555 to the one above would be:

DENY      alice                     take ownership
DENY      alice                     read, take ownership (Object inherit, inherit-only)
ALLOW   charlie                  read, execute/traverse
ALLOW   charlie’s group     read, execute/traverse
ALLOW   Everyone             read, execute/traverse
ALLOW   bob                      read, write (Inherit-only)

What happened here? The rights have been changed to reflect the requested mode, 555. Notice also that there was a DENY entry on the file owner in the previous QACL that no longer exists because it contradicted the chmod.

Now let’s look at the other ACEs. Why are there now two ACEs for alice?

DENY     alice                    read, take ownership (Object inherit)

became

DENY    alice                     take ownership
DENY    alice                     read, take ownership (Object inherit, inherit-only)

The first ACE preserves the permission denied on a non-POSIX right in the original ACE (take ownership is an SMB-only concept), and the second one is just a copy of the first one, marked inherit-only so that it can be passed onto children files and directories without affecting the current file. Because bob’s ACE was inherit-only to begin with, it doesn’t affect the permissions in this file anyway, and remains the same after the chmod.

The Bottom Line

Cross Protocol Permissions is a permissions mode, but what that really means is that it’s a collection of lots of decisions that the Qumulo team made in an attempt to best suit the needs of Qumulo’s users. There’s no perfect answer when it comes to this problem, but by using ID Equivalence, choosing not to hide existing access in the mode bits, and selectively affecting QACLs on changing the POSIX mode and on create, Cross Protocol Permissions provides an option to our users that is simple, requires no configuration, and just works.

Share with your network

GET A DEMO