Skip to main content

Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting us. A member of our team will be in touch shortly. Close

  1. Blog
  2. Article

Tom Haddon
on 21 April 2015

Giving developers production access without revealing secrets


It’s a classic trade off in the devops world:

On the one hand you want to give developers access to production systems so that they can see how their services are running and help debug problems that only occur in production.

On the other hand, these are production services. As such they necessarily have access to production “secrets” like SSL certificates, database passwords, or other data that you don’t want to share with anyone unless you absolutely have to.

So how do you balance these two conflicting priorities?

This is complicated even further by the fact that to have useful access from a developer perspective it needs to be as certain role accounts that their services run under. There’s no point giving them access to an unprivileged account that can’t see what it needs to be able to diagnose issues. But the role account itself necessarily has access to the secrets you want to restrict access to, otherwise it couldn’t provide the services in question.

Using an Apparmor profile to confine access

Here at Canonical we decided to solve this problem by giving developers access to connect to the servers in question via an unprivileged account (their own user account), and then giving those users sudo to the role account in question, but confined by an apparmor profile.

This apparmor profile would have all the access the role account does, but with read-only permissions, and with files known to contain secrets not even readable.

So how do we achieve this?

We’re heavy users of Juju so the natural choice for us was to write a subordinate charm that we could deploy to any servers and services we needed. Juju is a service orchestration tool that focuses on the relationships between services. A Charm encapsulates how a service is defined. Subordinates are services with knowledge of, and access to, their principal services.

Let’s take a look at how you’d deploy our confined-role-account subordinate charm in practice. In our example, we’re going to deploy MediaWiki and MySQL, manually add an unprivileged user account on the MediaWiki server, and then configure the confined-role-account subordinate charm so that our unprivileged user can sudo to the www-data user, but with restricted access.

Example deployment with MediaWiki and MySQL

First of all, configure your juju environment and fire up the bootstrap instance.

Then, deploy your charms into the environment:


juju deploy cs:charms/trusty/mediawiki
juju deploy cs:charms/trusty/mysql
juju add-relation mysql mediawiki:db

Now let’s manually create our unprivileged user. In a production environment we’d likely have a different means of doing this, such as via ldap.


juju ssh mediawiki/0
$ sudo adduser service-user

Now we can confirm that we can login as the account in question and see what sudo permissions it has by default:


juju ssh mediawiki/0
$ sudo su - service-user
$ sudo -l
Sorry, user service-user may not run sudo on juju-env-machine-1.

Now we can deploy the confined-role-account subordinate charm, set our config options for it, and add a relation to the mediawiki charm:


juju deploy cs:~canonical-sysadmins/trusty/confined-role-account
juju set confined-role-account group=service-user role-account=www-data extra-apparmor-rules=” audit deny /etc/mediawiki/AdminSettings.php r,”
juju add-relation mediawiki confined-role-account

What is this doing? It’s saying that anyone in the service-user group can sudo to the www-data role account with the default read-only permissions, but also with /etc/mediawiki/AdminSettings.php not visible to them. This file contains the database password, and in our scenario we want to allow developers access to the mediawiki server as the www-data role account without giving them access to the production database password.

In future we plan to modify the confined-role-account charm so that extra apparmor rules can be passed over the relation with the primary charm. This will allow charms to be opinionated about what secrets they should restrict access to, so no extra configuration needs to be set.

And now let’s confirm how things look:


juju ssh mediawiki/0
$ sudo su - service-user
$ sudo -l
Matching Defaults entries for service-user on juju-canonistack-machine-1:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin

User service-user may run the following commands on juju-canonistack-machine-1:


(www-data) NOPASSWD: /usr/local/sbin/apparmor-wrapper-service-user-www-data \"\"
$ sudo -u www-data /usr/local/sbin/apparmor-wrapper-service-user-www-data
$ echo “this is a test” > ~/this-is-a-test.txt
bash: /var/www/this-is-a-test.txt: Permission denied
$ head /etc/mediawiki/LocalSettings.php
$path = array( $IP, "$IP/includes", "$IP/languages" );
set_include_path( implode( PATH_SEPARATOR, $path ) . PATH_SEPARATOR . get_include_path() );

require_once( "$IP/includes/DefaultSettings.php" );

# If PHP's memory limit is very low, some operations may fail.
# ini_set( 'memory_limit', '20M' );

if ( $wgCommandLineMode ) {
$ head /etc/mediawiki/AdminSettings.php
head: cannot open '/etc/mediawiki/AdminSettings.php' for reading: Permission denied

We can also see our command history in /var/log/apparmor-wrapper/www-data-history.log:


echo "this is a test" > ~/this-is-a-test.txt
head /etc/mediawiki/LocalSettings.php
head /etc/mediawiki/AdminSettings.php

And there you have it. How to give developers production access without revealing secrets. So how do you implement this for yourself? You can deploy the charm with the following command:


juju deploy cs:~canonical-sysadmins/trusty/confined-role-account

Then just set any extra apparmor rules you need, and relate to any primary charm you want based on the example above.

Related posts


Michelle Anne Tabirao
13 December 2024

How does OpenSearch work?

Data Platform Article

How does opensearch work? OpenSearch is an open-source search and analytics suite. Developers build solutions for search and more! ...


Michelle Anne Tabirao
13 December 2024

What is RAG?

AI Article

RAG explained: is a technique that enhances generative AI models by utilizing external knowledge sources such as documents and extensive databases. ...


Matthew de Klerk
12 December 2024

What is vulnerability management?

Security Article

Vulnerability management is the holistic process of identifying and handling security risks in an organization’s networks, systems and devices. Vulnerability management serves an overarching strategy that describes and outlines the many individual efforts and steps taken to reduce cyber incident risk to acceptable levels and improve overa ...