In the first post in this series, we packaged up python bindings for the Qumulo API and uploaded them to AWS Lambda as a layer. Now, let’s use this layer!

Let’s make a lambda function to rotate the admin password of the cluster. We’ll use AWS Secrets Manager, a service to manage and store credentials, to rotate our Qumulo’s password. Leveraging this service and the Qumulo API, we can schedule recurring, automatic changes to the Qumulo’s admin password. Other services (one which we will write in a future post) can then access the current admin password via the AWS Secrets Manager API.

This cookbook uses AWS Lambda, IAM, and Secrets Manager, and a Qumulo Cluster running in AWS. To get started, gather the following:

  • The admin password of your Qumulo cluster
  • An IP address for the Qumulo cluster
  • The subnet ID and security group ID containing the Qumulo cluster
  • The ARN of the lambda layer created in the first post

We assume some variable names and that this is being run in us-west-2 region. Adapt these values for your environment.

First, let’s create a secret in AWS Secret Manger to store the Qumulo cluster’s credentials. Replace strings in carrots with values from your environment.

1. Fill in the blanks and run this command:

aws secretsmanager create-secret --name "my-qumulo-credentials" --description "credentials for my qumulo cluster" --secret-string '{"username":"admin", "password":"", "host":"[ip of a node in the cluster]"}'

2. Note the secret’s ARN from the output for later reference.

We will create a Lambda function to manage this secret in a moment. Before doing so, we need to create an IAM role that the Lambda function will assume. We’ll attach permissions to this role as we go.

1. Run this command:

aws iam create-role --role-name QumuloSecretRotationRole --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{"Action": "sts:AssumeRole", "Effect": "Allow", "Principal": {"Service": ""}}]}'

2. Note the ARN of the created IAM role for later reference.

The Lambda function will need access to AWS CloudWatch to upload logs and the Qumulo cluster’s VPC to change the admin password. The role “AWSLambdaVPCAccessExecutionRole” is perfect for this situation.

1. Run this command:

aws iam attach-role-policy --role-name QumuloSecretRotationRole --policy-arn "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"

Create the lambda function from the sample on the Qumulo Github. We configure the Lambda function to use the Lambda Layer created in the first post. The function implements Secret Manager’s rotation workflow.

1. git clone
2. cd lambda
3. zip ./ ./
4. Run this command:

aws lambda create-function --function-name "QumuloAdminPasswordRotationFunction" --runtime "python2.7" --handler "qumulo_secret_rotation_lambda.lambda_handler" --zip-file fileb:// --layers "[Qumulo API Layer ARN]" --vpc-config "SubnetIds=[Qumulo's subnet],SecurityGroupIds=[Qumulo's security group]" --timeout 30 --description "Rotate a Qumulo Cluster's Admin password" --publish --environment "Variables={SECRETS_MANAGER_ENDPOINT=}" --role "[IAM role ARN]"

5. Note the lambda function ARN for later reference.

The Secrets Manager service must be given permission to invoke our Lambda function. Run this command:

aws lambda add-permission --function-name "[lambda function ARN]" --statement-id SecretsManagerInvocation --principal "" --action "lambda:InvokeFunction"

Now that we’ve created a Lambda function, we can grant it permission to change the Qumulo secret. We want users of this function to only be able to change the secret via the function, which we accomplish by specifying the ARN of the function in the policy. After this step, all the needed permissions to call our lambda function and have it do its job should be configured.

1. Edit qumulo-lambda-samples/qumulo_secret_rotation_policy.json by replacing with the one noted above.
2. Run this command:

aws iam put-role-policy --role-name QumuloSecretRotationRole --policy-name "QumuloSecretRotationPolicy" --policy-document file://qumulo_secret_rotation_policy.json

Now that we have a secret, function, and permissions configured, we can associate the lambda function with the secret and enable automated rotation.

Run this command:

aws secretsmanager rotate-secret --secret-id "" --rotation-lambda-arn "[lambda function ARN]" --rotation-rules "AutomaticallyAfterDays=30"

A rotation was triggered automatically. Give it a minute to finish and then check secrets manager to see the new password (or CloudWatch logs to see any issues). This command will retrieve the secret:

aws secretsmanager get-secret-value --secret-id "[secret ARN]"

Now your Qumulo password is in a safe place and changed regularly! In the next post, we will use S3 notifications and the Qumulo API to write a Lambda function that copies files to Qumulo when they appear in an S3 bucket. We’ll make use the secret created here to programmatically log into the cluster to upload files.