Qumulo ships a new version of our Qumulo Core software approximately every two weeks, and with each version of Qumulo Core we release a new version of the Qumulo REST API, along with a Python wrapper as a convenience for Python language developers. This is very useful for users of high-level programming languages such as Python or any other programming language that can make REST calls directly.

But what about system administrators and others that use bash/shell scripting for many of their tasks? How can they take advantage of the power of the Qumulo REST API? For these users, the Qumulo-provided qq command line (CLI) tool is the way to harness the power of Qumulo’s REST API from the CLI and in shell scripts. The entirety of Qumulo’s REST API is exposed via qq.

This post will show you some ways you can leverage information from your Qumulo cluster via qq directly in the shell or in bash scripts, with some examples showing how to integrate output from qq into other management scripts.

Installing qq

The simplest way to install qq is to install the Qumulo REST API from PyPI (the public Python Package Index repository):

https://pypi.python.org/pypi/qumulo_api/

To install, open a command prompt and then run:

pip install qumulo-api

You will need python 2.7 in order to run qq. If you prefer, you can also run qq locally on your cluster (via ssh) where it is pre installed.

Getting Started: Logging in and viewing cluster content

To start, you need to log in to your qumulo cluster, like so:

qq --host [hostname or IP address] login -u [username] -p [password]

Where username and password are credentials on your Qumulo cluster. On successful login, an access token is created behind the scenes, so while that token is valid, you do not need to specify username and password for subsequent qq commands.

To see all of the options from the command line, try running qq --help.

Once you’re authenticated, you can view information about cluster content…. This command lists the contents starting at the root directory:

qq --host music fs_read_dir --path /

Which returns content like this (a few thousand lines of output elided here….):

{
    "child_count": 131,
    "files": [
    (...)
        {
            "blocks": "1",
            "change_time": "2017-01-04T22:59:52.909017431Z",
            "child_count": 30,
            "creation_time": "2016-03-15T20:12:07.722559693Z",
            "datablocks": "0",
            "file_number": "3",
            "group": "17179869184",
            "group_details": {
                "id_type": "NFS_GID",
                "id_value": "0"
            },
            "id": "3",
            "metablocks": "1",
            "mode": "0777",
            "modification_time": "2017-01-04T22:59:52.909017431Z",
            "name": "users",
            "num_links": 30,
            "owner": "12884901888",
            "owner_details": {
                "id_type": "NFS_UID",
                "id_value": "0"
            },
            "path": "/users/",
            "size": "15360",
            "symlink_target_type": "FS_FILE_TYPE_UNKNOWN",
            "type": "FS_FILE_TYPE_DIRECTORY"
        }
    ],
    "id": "2",
    "paging": {
        "next": "",
        "prev": "/v1/files/%2F/entries/?before=4190957568&limit=16"
    },
    "path": "/"
}

This output illustrates a few things about qq and the Qumulo REST API:

  1. The output from qq commands is JSON format
  2. For some qq commands, the output can be very verbose

The unix philosophy is to “write programs to handle text streams”, and that is also the foundation of shell scripting in unix. While JSON format is text, it is more common to pipe either comma- or tab-separated values to subsequent commands. So to tailor the output from qq commands to be more terse and output different formats, for this blog post we will use the popular jq command-line JSON processor to illustrate how you can make qq fit in more readily with other CLI tools (on a Mac, you can install jq using homebrew; for linux, Windows and other flavors see the tool website).

Using the above example, let’s grab some of the output from qq and use jq to make it easier to read and repurpose in other tools (there is a tutorial for jq that shows you some of what you can do):

qq --host music fs_read_dir --path /media | jq '[.path, .id, .files[0].blocks] | @csv'

This will return the path, id and blocks value from the first entry returned by fs_read_dir in csv format, like this:

"\"/media/\",\"1000003\",\"1\""

Which could then be piped to other tools that expect CSV format. Note that you can also create files and directories using qq and the REST API, but the more common way is to mount shares using NFS or SMB and create content on the cluster that way.

Information about your Cluster, and viewing and creating Users, Groups and NFS or SMB Shares

Let’s say you want to retrieve the capacity and state of all of the slots (disks) in your cluster, order the results by slot, and then store the results in a csv file as part of an IT status report. You could run:

qq --host music cluster_slots | jq '[ .[] | {"cap":.capacity, "slot":.slot_type, "state":.state }] | sort'

You can create groups:

qq --host music auth_add_group --name "My Test Group"

{
    "gid": "",
    "id": "1026",
    "name": "My Test Group",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1026"
}
Expanded identity information for group 1026: [
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And add users to the group:

qq --host music auth_add_user --name "My_Test_User" --primary-group "My Test Group" -p junk

{
    "id": "1027",
    "name": "My_Test_User",
    "primary_group": "1026",
    "sid": "S-1-5-21-1203876804-525990034-1406135336-1027",
    "uid": ""
}
Expanded identity information for user 1027: [
    {
        "id_type": "LOCAL_USER",
        "id_value": "My_Test_User"
    },
    {
        "id_type": "LOCAL_GROUP",
        "id_value": "My Test Group"
    }
]

And then create a directory and add a share for the new group:

qq --host music fs_create_dir --name "my_test" --path "/"

With:

qq --host music smb_add_share --name "my_test_smb_share" --fs-path "/my_test/"

Or if you’re using NFS:

qq --host music nfs_add_share --fs-path "/my_test/" --export-path "/my_nfs_export" --no-restrictions

Performance: Retrieving Analytics and Aggregate Data

One of the most powerful aspects of the Qumulo REST API are the real-time analytics and aggregate data about the filesystem exposed via our REST API. Using qq this data is exposed to users using bash scripts as well.

Let’s say you want to retrieve all current activity in the cluster and display it in tabular form similar to iostat and other tools. To get current activity you can just run:

qq --host music current_activity_get | jq '.entries | sort_by(.ip)'

Like all qq commands this will return JSON-based data — in this case showing current cluster activity such as file and metadata throughput and IOPs — grouped by client IP address:

This data can then be piped to other tools such as monitoring tools.

Further Reading and tools

There is a good tutorial on JSON and the jq tool here.

If you are currently a Qumulo customer, you can find more information on the Qumulo Community site (search for ‘qq’ in conversations).

We’re engineers, builders, craftsmen and artists. We build products that are always on, that never lose data, and that store and retrieve data fast. Connect with us on social media!

Share with your network