We release our Android apps every day

How we setup Android apps continuous deployment

By Piotr Wilczek

Releasing a new version of an Android app can be a dreaded job for most Android developers. We think:

“Oh no, time for another release… I need to take care of translations, prepare the changelog, wait for the build, test the latest changes, and pray Proguard didn’t mess with things it shouldn’t. 😩”

'Are you stealing those LCDs?' 'Yeah, but I'm doing it while my Android app builds.'

Photo by Randall Munroe licensed under CC BY-NC 2.5

But the release should be the best part of our work! It’s the moment we share our hard work with our users. It’s the time we finally give them all those new shiny features and bugs fixes.

“Deployment Frequency” is also a DORA metric. The research shows that frequent releases have a positive correlation with team performance. Yes — mobile apps are distributed through third-party stores and developers don’t fully control the delivery process. But we can still try to push the borders.

This post explains how we transformed the release process of our Todoist and Twist Android apps from the old-fashioned multi-manual steps to brisk “continuous deployment”, and how that worked out for us. I hope you can also apply some of those improvements in your deployment process and make it more seamless and fun!

The old way of doing things

Before, our release process was quite complicated. We released infrequently, with many updates in each release. There were quite a few things to remember (and easy to forget) during the process. Below I’ll describe our most significant issues.

Big branches

The biggest issue we had was long-lasting branches with thousands of code line changes.

Whenever the developer wanted to implement a new feature, they opened a new branch and worked on it in separation till completion. Rebase operations had many conflicts and the review was long with much back-and-forth between author and reviewer.

All this could take months to complete for a moderate-sized feature, and that was against another key DORA metric — “Lead Time for Changes” — the amount of time it takes a commit to get into production.

Blocking translations

Todoist is translated into more than 15 languages and Twist into 8 languages. So when a release was planned, we first had to merge all related PRs, “freeze” the main branch, push strings for translations and wait a few days till all translators completed their work.

Building and Testing

Before R8 was introduced, we used Proguard with a heavily customized configuration that squeezed every last drop of performance, making a production build take close to 2 hours.

After that, we manually tested the app, and of course, it wasn’t uncommon to find new bugs. After fixing, we had to re-build — another 2 hours of waiting.

Fortunately, R8 decreased the build time to few minutes. This felt amazing! But we still had issues, and finding them just before the release (or even worse - after) was a nasty surprise.

The new way of doing things

As you can see, that was plenty of manual work. We even created a script that printed a runbook to help the developer follow it. But we didn’t stop there.

We’ve taken a high-level approach, examined best practices and made a plan. Then we’ve started fixing all issues and root causes one by one.

Here is how.

Feature flags

The change which had the biggest impact was introducing feature flags. If you’ve never used a feature flag, it’s a way to control the app behavior remotely without deploying a new version. In our apps, it usually looks like this:

This and the following code snippets have been simplified to keep them concise.

if (FeatureFlag.TASK_DESCRIPTION.isEnabled()) {
    showTaskDescriptionView()
} else {
    hideTaskDescriptionView()
}

Now, whenever we start working on a new feature, we set up a new feature flag and merge small pull requests into the main branch frequently. Because we keep the feature hidden from the public, it’s not a problem that it’s incomplete.

Another significant benefit is the ability to show a feature to a group of selected users and collect early feedback without building a custom APK.

Finally, it also allows us to gradually release a feature and quickly disable it if something goes wrong.

Automating translations

The second improvement we’ve made was automating the translation process.

The translation platform we use is Transifex , and for that, we’ve set up two GitHub Actions.

The first pushes new strings to Transifex whenever there is a change in one of our strings files on the main branch. This action also notifies our translation manager, who further communicates with the translation team and oversees the work. In short, it looks like this:

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  push:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2

      - name: Setup environment
        ...

      - name: Push strings to Transifex
        env:
          TX_TOKEN: ${{ secrets.TRANSIFEX_TOKEN }}
        run: tx push -s

      - name: Ping Localization team
        uses: Doist/twist-post-action@master
        with:
          message: "Todoist strings have been just pushed to Transifex. 🙂"
          install_id: ...
          install_token: ...

      - name: If failed, ping Android team
        if: ${{ failure() }}
        uses: Doist/twist-post-action@master
        with:
          message: "❌ Transifex push action failed"
          install_id: ...
          install_token: ...

The second action runs twice a week, and it automatically pulls translations from Transifex.

It also uses our Android Translations Check Gradle Plugin to verify if translations match the original strings format, so we don’t have to check them manually.

The action runs the task and then opens a pull request:

on:
  schedule:
    - cron: '0 0 * * 2,4'
  workflow_dispatch:

jobs:
  pull:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2

      - name: Setup environment
        ...

      - name: Pull strings from Transifex
        env:
          TX_TOKEN: ${{ secrets.TRANSIFEX_TOKEN }}
        run: tx pull -f -a --parallel

      - name: Check translations
        run: ./gradlew checkAndroidTranslations

      - name: Open a PR
        uses: peter-evans/create-pull-request@v3
        ...

      - name: If failed, ping Android team
        if: ${{ failure() }}
        ...

Tip: Feature flags can be used for parts of the app that still need translations. How? For example, when we implement a new dialog with some text, we hide it behind the feature flag and push it to the main branch. The strings are automatically sent for translations, and when they’re translated, the only thing we have to do is to enable the flag (or remove it).

Automating the changelog

To automate the changelog, we’ve written and open-sourced Changelog Gradle plugin to help us manage it. Our changelog consists of a CHANGELOG.md file that includes changes introduced with every public release and a changelog directory with unreleased changes. They’re structured in files named after their branch and contain a line or two describing the changes.

CHANGELOG.md
changelog/
├─ new-feature-1-branch.txt
└─ new-feature-2-branch.txt

We also have a GitHub Action that runs the Gradle task to validate the entries and ensure they follow our format. The plugin configuration is:

changelog {
    pendingChangelogDir.set("changelog")
    changelogFile.set("CHANGELOG.md")

    addRule("max length is 72 characters") { it.length <= 72 }
    addRule("cannot end with a dot") { !it.endsWith(".") }
    addRule("should start with a ⭐ or 🐛 or 🕶") {
        it.startsWith("⭐ ") || it.startsWith("🐛 ") || it.startsWith("🕶 ")
    }

    commit {
        val date = LocalDate.now().format(DateTimeFormatter.ISO_DATE)
        prefix = "## $date"
        postfix = ""
        entryPrefix = "- "
        insertAtLine = 2
    }
}

During the release, we fire another task from our plugin. It collects all pending entries from the changelog files and appends them to the CHANGELOG.md file together with a link to the PR which introduced the given change.

This changelog gets automatically published to a Twist channel where we share our releases. Later, our copywriters update the public Google Play changelog accordingly.

Version names

Lastly, we automated version names. We were using SemVer before, but it’s hard to automate and doesn’t make sense with continuous deployment.

Why?

When we release tiny changes multiple times per day, every release feels just like a patch. Also, with feature flags, there is no specific version that introduces new functionality so MAJOR.MINOR.PATCH scheme cannot be applied anymore.

To solve that, we’ve switched to simple incremental numbers matching version codes and prepended with v, for example, v8290. We then tag commits with the release names, and we have Version name Gradle plugin that assigns version name and version code in our gradle.build.kts file based on the latest git tag:

plugins {
    id("com.doist.gradle.version-name")
}

android {
    defaultConfig {
        versionCode = project.versionCode()
        versionName = project.versionName()
        ...
    }
    ...
}

Gradle Play Publisher

It’s important to mention that we also started to use Gradle Play Publisher by Alex Saveau that adds Gradle tasks to publish bundles on Play Store. That’s not a trivial project, and the code quality is very high. Thanks Alex!

Reaping the benefits

The above changes took us some time to implement. Most of them had an immediate impact, but only together they allowed us to start making continuous deployments.

Automated internal releases on every merge

To kick-off, we have a GitHub Action that makes an internal release whenever the main branch changes. It tags the latest commit with a new version name, builds the bundle file, and publishes the release in the internal track on Play Store. No magic here.

Weekly public releases with a simple trigger

Our favorite GitHub Action is the one that makes public releases. It’s quite complex, but we’ve split the process into Gradle tasks and wrapped them in a Gradle plugin. We reuse it across our projects, and we can also run the release locally in case GitHub Actions are down.

To be specific, we have two GitHub Actions for public release and one manual step.

Preparing the release

This action is fired manually. First, it runs our prepareRelease Gradle task, which does few things:

  1. it creates a release branch,
  2. merges the pending changelog into the main CHANGELOG.md file,
  3. runs git commit,
  4. finally, it tags the commit with the next version name.

After that, the action builds bundle and apk files and fires the second Gradle task draftRelease. This Gradle command does the following:

  1. it pushes the branch and tags,
  2. opens the PR,
  3. drafts the GitHub release.

Testing

After the first action finishes, we install the apk on our devices and do the last manual smoke test.

Even though we could fully automate the public release process, as we did with internal releases, we decided that we still want to have a final check done by a human before publishing the release. It’s a reasonably low-effort task, so it’s not a big problem. But we may drop this in the future and have fully automated public releases as well.

Release

If all works fine, we approve the release PR. That fires the second action, which publishes the bundle files on Play Store and runs one last Gradle task finalizeRelease that:

  1. publishes GitHub release draft,
  2. merges release PR,
  3. announces the release in our Twist release channel.

At this point, the release is finished, and the developer can happily go back to coding. 🙌

What’s next

Even though our release process is vastly improved, we’re still looking for ways to make it better.

One idea is dropping the manual test and making the release process fully autonomous, as mentioned before.

Another is automating the rollout itself. Right now, our initial rollout is 2%. At this point, we keep an eye on the crash rate and reported bugs. Then, if all looks good, we gradually increase the rollout to 10% and later to 100%. We are planning to automate this soon by periodically firing a GitHub Action that checks the stability of a release, and based on some conditions, increases the rollout.

Also, I haven’t mentioned anything regarding automated tests. We don’t run our test suite during the release. We run tests on every PR, and assume that tests on the main branch pass but of course, in practice, it’s not always the case. There are also issues with flaky tests, but that’s a pandora box I don’t want to open in this blog post.

So yes, another improvement we could make is a step that runs UI tests against the release build.

Result

In the end, we’re as close to continuous deployments on Android as Google allows. Needless to say, we’re delighted with the results.

We’ve got rid of a scary two-hour manual process and replaced it with an automated setup based on Gradle tasks and GitHub Actions.

As a result, we follow universal best practices, release smaller changes much more frequently, and get value in the hands of our users faster.