TPS-2019-002 SSH public keys of Joyent users/development hosts in some published images

Overview

In the process of creating images, some of Joyent’s internal-use SSH public keys were inadvertently left in certain published images. This led to the risk of potential unauthorized access to instances using the affected images.

Joyent acknowledges the assistance of an Open Source user in discovering this issue.

Background

Joyent creates and publishes images to our Triton public cloud. These images are of various operating systems, to be used by customers in creating instances that run on the cloud.

We also make these images available to customers running Triton in their own datacenters, and to stand-alone SmartOS users.

Description

An Open Source user reported to Joyent that certain images contained some unexpected public key data. Joyent investigated and confirmed that – due to a new bug in the image-building system – the following images were not properly stripped of SSH public keys used during the development process and still contain these keys in the /home/ubuntu/.ssh/authorized_keys file:

  • ubuntu-certified-16.04-20190122 81c0ef69-e9d7-4e93-a15b-efd7ea9c9ee8
  • ubuntu-certified-18.04-20190122 c9db249c-93ba-4507-9fa4-b4d0f81265fc

As part of the instance creation process, these keys would be propagated to the running instances that were using these images. These keys would make it possible for the Joyent developers (and Joyent development systems with the corresponding private key) to access these customer systems.

To the best of our knowledge, no such access was ever attempted in any systems using these images.

Actions Taken by Joyent

Upon discovery:

  • The images listed above were disabled in our Triton public cloud, so that no further instances could be created from them.
  • An audit was conducted to inventory all Triton public cloud instances created from the affected images, and all affected public cloud customers will be notified within hours of this advisory’s first posting.
  • As a good-faith measure, the applicable SSH keys of Joyent employees/systems were all rotated out.

Actions You Need to Take

Triton Enterprise Software Users and Triton Public Cloud Users

The keys left in the image have the following fingerprints:

md5

f9:41:4a:eb:84:c1:28:5e:31:27:c7:e8:c8:46:84:27
e2:2e:1d:40:d2:06:1d:2b:e1:27:3e:af:e8:f9:95:d9
35:35:c6:9a:f8:f3:d3:46:b7:4c:d9:9d:82:6b:3d:60
09:46:06:e8:9c:58:94:cb:8e:f2:7e:bf:a6:2b:a8:76
6f:7b:b7:f6:f4:1a:a1:28:70:fb:33:03:ec:04:fb:b5
83:8b:80:66:7a:24:d5:44:81:b3:eb:4a:eb:31:b5:b1
53:6e:35:43:1a:8a:b9:ec:36:1e:12:20:70:6b:69:76
57:4f:70:c6:bb:b6:2b:53:bc:71:aa:64:cd:30:5e:a7
7e:e0:00:29:fc:5e:5b:89:b3:03:71:12:0e:43:53:00
aa:f6:5e:c9:bf:9a:72:8c:cc:04:13:a1:e5:14:8f:3b

sha256

SHA256:scjD+2EYAJRz8wBreM9iXCk0oJ4NKTvTy2NgzlZG8JQ
SHA256:0dR6/B605RKNpw8SZbAU6fcvTH7ylXQW079QeygaBDM
SHA256:EMBapnoGdvBYoFLu912LHBG3eBoaGKMl8mq3wY5x5pI
SHA256:tWhV06pPcS0A78tb5PBeWQ1cG27GUIDfmuX6/ze36tU
SHA256:sxr9LU+d+8+y5YI6i6U8mGeT6/DKmEnI/g8SyC+XHs8
SHA256:+gcoJKBMgUbtSaA0xTSdzNY48Qr8zx0C93mjX7NV5Tw
SHA256:XmQD/GojiNxYVATLspPwa2Zx5hdb4+iyFWHXcHhugIw
SHA256:qaWrGQrhbYDQHkVK3J5AEHt3QEXlReWDgei4m29cz04
SHA256:Kp1Vo58n2uMup1NQwzQ8ympSTsiO2iFylpvPBiOVy5Y
SHA256:XQ7XuhLmsCx04uthU8z+GhFZB4xiE+Lh+qsFzNNaOeY

If you have any non-deleted instance(s) created from the images listed in the “Description” part of the Overvew above (running in the Triton public cloud, or in your on-premises datacenter), proceed as follows:

  • Check your instances to ensure that any user’s .ssh/authorized_keys file contains all of the keys for which you control the matching private key, and only those keys. If an instance’s file contains any other keys with fingerprints matching the above list, it may have been affected.
  • Affected instances should be destroyed and recreated using unaffected images, to ensure that unauthorized Joyent SSH keys are not present. Note: Before destroying an instance, back up any data you wish to retain.

Open Source Triton Users

Please direct any further questions to The SmartOS Community Mailing Lists and IRC.

Support

If you are a Joyent customer and have any further questions or concerns after reading the information provided above, please contact Joyent Support.

As noted above, if you are an Open Source Triton user, please direct any further questions to the SmartOS Community Mailing Lists and IRC.