Cloud Workstations: building reusable development environments in cloud — Part 2

How to customize your Cloud Workstations environment and improve developer productivity.

In the first part we reviewed the pros and cons of local vs cloud development environments and checked how to create a minimal yet fully functional setup of the Google Cloud Workstations using Pulumi.

Cloud Workstations: building reusable development environments in cloud — Setting up Cloud Workstations using Pulumi, TypeScript and Bun

Now it’s time to make that development environment yours and see how one can customize base images and ensure developers have all the tools they need.

Defining a customized base image

GCP provides a set of base images available out of the box that provide a set of pre-configured IDEs — named Code OSS and IDEs from the JetBrains toolbox (IDEA, PyCharm, WebStorm, etc.).

Preconfigured IDEs | Cloud Workstations | Google Cloud Documentation — Preconfigured IDEs for use with Cloud Workstations

You can check out the complete list of the IDEs and their respective docker images here.

Code OSS provides a ready-to-use web UI while JetBrains IDEs expose a remote development environment gateway to which you can connect from your local copy of the IDE. Both approaches have their pros and cons but why don’t we just combine them together?

To create our customized base image for the workstation instance we will use Code OSS and JetBrains WebStorm base images together to provide an easy-to-start IDE with terminal and file explorer in your web browser with Code OSS and also expose JetBrains gateway to connect your local IDE.

Available extension points

The core part of the workstations base images setup is distinction between system and user space. While workstations work with a containerized environment, your IDE is actually running inside a container with a volume mounted in it to e.g. your /home directory. It means that when you install some tooling or perform container configuration, you should account for that as well.

Depending on the tooling you are going to install and whether you want to maintain the ability to update it and configure it from within your user space, you may want to move some tooling installation scripts from the Docker image setup to user space scripts.

On startup, base images run files under /etc/workstation-startup.d/* in lexicographical order to initialize the workstation environment. There is a special script in that folder called 030_customize_environment.sh that executes /home/user/.workstation/customize_environment as user.

You can unwrap the complete entrypoint setup by examining /google/scripts/entrypoint.sh in the base images.

Base image Dockerfile

The absolutely minimal customized setup with Code OSS and WebStorm looks like this:

FROM us-central1-docker.pkg.dev/cloud-workstations-images/predefined/code-oss:latest AS code-oss-image

FROM us-central1-docker.pkg.dev/cloud-workstations-images/predefined/webstorm:latest AS runtime

COPY --from=code-oss-image /opt/code-oss /opt/code-oss
COPY --from=code-oss-image /etc/workstation-startup.d/110_start-code-oss.sh /etc/workstation-startup.d/110_start-code-oss.sh

Now you can sprinkle it with some standard tooling, e.g. add mysql and redis clients, maybe gh and lefthook CLIs — these are the ones that we just want to bundle into the image.

RUN apt update \
    && DEBIAN_FRONTEND=noninteractive apt install apt-utils software-properties-common unzip -fyq \
    && apt upgrade -fyq \
    # Install MySQL client \
    && DEBIAN_FRONTEND=noninteractive apt install mariadb-client -yq \
    # Install Redis Client \
    && curl -fsSL https://packages.redis.io/gpg | gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg \
    && chmod 644 /usr/share/keyrings/redis-archive-keyring.gpg \
    && echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/redis.list \
    && apt update \
    && DEBIAN_FRONTEND=noninteractive apt install redis -yq \
    # Install GitHub CLI \
    && mkdir -p -m 755 /etc/apt/keyrings \
    && out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg \
    && cat $out | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
    && chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
    && mkdir -p -m 755 /etc/apt/sources.list.d \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
    && apt update \
    && DEBIAN_FRONTEND=noninteractive apt install gh -yq \
    # Install lefthook \
    && curl -1sLf 'https://dl.cloudsmith.io/public/evilmartians/lefthook/setup.deb.sh' | sudo -E bash \
    && DEBIAN_FRONTEND=noninteractive apt install lefthook -yq

The next step is to add some pre-configured IDE extensions. This is probably specific to your team’s needs. JetBrains plugins can be installed as follows:

# Install .env - https://plugins.jetbrains.com/plugin/9525--env-files
RUN bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    9525 \
    # Install .ignore - https://plugins.jetbrains.com/plugin/7495--ignore
    && bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    7495 \
    # Install google cloud code - https://plugins.jetbrains.com/plugin/8079-google-cloud-code
    && bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    8079 \
    # Install Gemini Code Assist - https://plugins.jetbrains.com/plugin/24198-gemini-code-assist
    && bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    24198 \
    # Install JetBrains AI Assistant - https://plugins.jetbrains.com/plugin/22282-jetbrains-ai-assistant
    && bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    22282 \
    # Install JetBrains Junie - https://plugins.jetbrains.com/plugin/26104-jetbrains-junie
    && bash /installer-scripts/plugin-installer.sh \
    -d /opt/WebStorm/plugins/ \
    26104

The logic is simple here: you open up the plugin page (e.g. https://plugins.jetbrains.com/plugin/26104-jetbrains-junie), copy the ID that goes after /plugin, and install it with the plugin-installer.sh script.

For Code OSS we can either download extensions manually during the build phase or prepare a user-space script and use /opt/code-oss/bin/codeoss-cloudworkstations --install-extension. Let’s prepare a user-space script 120_install_vs_code_extensions.sh:

#!/usr/bin/env bash

sudo -u user /opt/code-oss/bin/codeoss-cloudworkstations \
  --install-extension ms-azuretools.vscode-containers \
  --install-extension pulumi.pulumi-vscode-tools  \
  --install-extension GoogleCloudTools.cloudcode \
  --install-extension Google.geminicodeassist \
  --install-extension cweijan.vscode-database-client2 \
  --install-extension cweijan.dbclient-jdbc \
  --install-extension RooVeterinaryInc.roo-cline \
  --install-extension redhat.vscode-yaml \
  --install-extension dbaeumer.vscode-eslint \
  --install-extension orta.vscode-jest \
  --install-extension gamunu.vscode-yarn

And then copy the script with:

COPY 120_install_vs_code_extensions.sh /etc/workstation-startup.d/
RUN chmod +x /etc/workstation-startup.d/120_install_vs_code_extensions.sh

customize_environment.sh

Another alternative solution is to create a customize_environment script in the user’s home folder. Let’s add pulumi and nvm to the setup. Those tools are usually installed into the user’s home.

Let’s ensure the home directory will be available first and create 011_customize_user.sh:

#!/usr/bin/env bash

sudo mkdir -p /home/user/.workstation/
sudo mkdir -p /home/user/.local/bin/
sudo cp /tmp/customize_environment.sh /home/user/.workstation/customize_environment

sudo chown -R user /home/user/
sudo chmod +x /home/user/.workstation/customize_environment

Now let’s create customize_environment.sh where we install Pulumi, NVM, some LTS node and Gemini CLI.

#!/usr/bin/env bash

set -x;
export DEBIAN_FRONTEND=noninteractive
export PULUMI_VERSION="3.206.0"
export NVM_VERSION="0.40.3"

curl -fsSL https://get.pulumi.com | bash -s -- --version "${PULUMI_VERSION}"

# This is required for some tools that source from local bin but ignore bashrc
sudo ln -s ~/.pulumi/bin/pulumi ~/.local/bin

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

nvm install --lts
npm install -g corepack @google/gemini-cli

Now we need to add those two scripts to the base image with:

COPY 011_customize_user.sh /etc/workstation-startup.d/
COPY customize_environment.sh /tmp/customize_environment.sh
RUN chmod +x /etc/workstation-startup.d/011_customize_user.sh \
    && chmod +x /tmp/customize_environment.sh \

We’re all set. We have explored three ways to customize a Cloud Workstation base image:

• by extending the base image and installing tooling into the image itself
• by providing workstation startup scripts
• by creating a customize_environment script in the user’s home folder

You can find a complete example of the WebStorm + Code OSS base image in the xSAVIKx/gcp-cloud-workstations-howto repository.

gcp-cloud-workstations-howto/customized/base_images/webstorm at main — Example infrastructure setup for GCP Cloud Workstations using Pulumi

Letting Pulumi do the job

Now that we’re all set from the base image perspective, we need to fine-tune our Pulumi infrastructure setup.

You can jump to a complete setup and check it out here.

Artifact Registry setup

We will be using Google Artifact Registry to store our customized image as this simplifies authentication for the Workstations, keeps the image physically closer to the virtual machine, as well as provides fine-grained control over the private images.

First, we need to enable Artifact Registry and legacy Container Registry services using the enableServices method we created before or just by extending the requiredServices array with:

"artifactregistry.googleapis.com",
"container.googleapis.com",

Now here’s how you can define a Docker registry using Artifact Registry service:

function defineArtifactRegistry() {
  const artifactRegistry = new gcp.artifactregistry.Repository(
    "dockerRegistry",
    {
      location: region,
      repositoryId: "containers",
      description: "Private containers registry",
      format: "DOCKER",
      dockerConfig: {
        // usually better set to `true`, but for the lab we're setting it to false
        // to ease re-creation of the same containers.
        immutableTags: false,
      },
    },
    { provider: gcpProvider, dependsOn: services },
  );
  return { artifactRegistry };
}

Finally, we’ll create a helper method to grab an authentication token to the registry as it’s private and secure by default:

async function accessToken() {
  const auth = new GoogleAuth({
    scopes: ["https://www.googleapis.com/auth/cloud-platform"],
  });
  return (await auth.getAccessToken()) || undefined;
}

Creating the Docker image

In order to run docker build from Pulumi, we’ll need to add Pulumi Docker provider and define Image resource.

First, adding a provider with bun add '@pulumi/docker' and now defining an image as follows.

const webstormImage = new docker.Image("webstormImage", {
  build: {
    context: `${__dirname}/base_images/webstorm`,
    dockerfile: `${__dirname}/base_images/webstorm/Dockerfile`,
    platform: "linux/amd64",
  },
  imageName: pulumi.interpolate`${artifactRegistry.registryUri}/webstorm:latest`,
  registry: {
    server: artifactRegistry.registryUri,
    username: "oauth2accesstoken",
    password: await accessToken(),
  },
  skipPush: false,
});

We’re using Dockerfile we created before and Artifact Registry as a destination for our image. And also authenticating our code to access the registry using a short-term OAuth2 access token.

Preparing the Workstation configuration

With the image ready, you can add container.image property to the WorkstationConfig to override the base image. We will also configure the GCE host with a specified machine type and home disk. And to finalize the setup we’ll configure automatic VM idle and run time — this will allow us to save some costs when the machine is not actively used.

const wsCustomizedConfig = new gcp.workstations.WorkstationConfig(
  "wsCustomizedConfig",
  {
    workstationConfigId: "customized-config",
    workstationClusterId: wsCluster.workstationClusterId,
    location: region,
    container: {
      image: webstormImage.repoDigest,
    },
    idleTimeout: "3600s",
    runningTimeout: "43200s",
    host: { gceInstance: { machineType: "e2-standard-4" } },
    persistentDirectories: [
      {
        mountPath: "/home",
        gcePd: {
          diskType: "pd-standard",
          sizeGb: 200,
          reclaimPolicy: "DELETE",
        },
      },
    ],
  },
  { provider: gcpProvider, dependsOn: services },
);

Summary

We are all set. Running pulumi up and in a bit of time we have our own Code OSS + WebStorm setup with a fully customized base image that your team can reuse easily without having to think twice about which version of the tool they had and if there’s anything else they missed.

We have reviewed three options for customizing Cloud Workstations base images. We prepared a custom base image with Code OSS and WebStorm support and updated our Pulumi setup to automatically build Docker image and update our workstations configuration.

In the next part we will cover the networking and security part of the setup.

Cleanup

If you want to delete the resources Pulumi created, just run pulumi down and it will take care of the cleanup.

Useful resources

• GCP Cloud Workstations How To repository
• Cloud Workstations home page
• Pulumi GCP provider Workstations page
• Pulumi Docker provider
• Artifact Registry