App KPIs

Part
01
of one
Part
01

App KPIs

According to a developers' documentation guide on developing "games for all screens," some potential KPIs or best-practice metrics related to game performance for cold, warm, and hot startups include monitoring cold startup interval (should be less than five seconds), warm startup interval (less than two seconds), and hot startup interval (less than 1.5 seconds). Other metrics include monitoring Play Store Android Vitals, crash rates, and ANR (App Not Responding) instances.

COLD STARTUP INTERVAL

  • A cold start refers to a situation when an app starts from scratch. That is, the systems process had not before this start, created any of the app processes.
  • Cold starts usually occur whenever an app is being launched for the first time after a device is booted, or after the system has terminated/killed the app processes.
  • It is vital to monitor cold start intervals because this type of start "presents the greatest challenge" when minimizing startup time is considered. This is because the system and app are required to do more work when compared to other launch states (such as warm or hot start.)
  • As a potential KPIs and best practice metric when developing "games for all screens," cold start should not take more than five (5) seconds.
  • According to a developers' documentation guide on developing "games for all screens," a system performs three tasks during cold startup including (1) loading and launching of the app. (2) Displaying a blank "starting window for the app" after launch, and (3) creating the app process. The best practice (KPI interval) is that these three processes should not take more than five seconds.
  • It is vital to monitor startup interval because app users expect games to be "responsive and fast to load." An app or game that has a slow startup time does not meet this user expectation and will be disappointing to them.
  • A poor game startup experience can cause users to rate an app poorly using the Play store or completely abandon an app altogether. This experience directly relates to bounce rates, unfavorable return rates, and high uninstall rates.

WARM STARTUP INTERVAL

  • According to the best practice published by within a developers' documentation guide on developing "games for all screens," the ideal warm startup interval should be less than 2 seconds.
  • A hot start of an application is more straightforward and consumes lower-overhead when compared to a cold start. During hot-starts, the system brings the activity of a user to the foreground. If all the user's application activities are still contained in device memory, the app avoids repeating "object initialization, layout inflation, and rendering."
  • Often, some memory gets purged when a system responds to memory trimming events. Memory trimming events include onTrimMemory(); in such instances, objects will have to be recreated "in response to the hot start event."
  • Hot starts exhibit similar on-screen behavior when compared to a cold start scenario: The system process usually displays a blank screen pending when the app finishes rendering the activity.
  • Whether some memory has been purged or not, the best practice published by within a developers' documentation guide on developing "games for all screens," recommends an interval less than two seconds as the KPI for an acceptable warm startup.

HOT STARTUP INTERVAL

  • According to best practices published by within a developers' documentation guide on developing "games for all screens," the ideal hot startup interval should be less than two seconds.
  • A hot start of an application is less complicated and consumes lower-overhead when compared to a cold start.
  • During the hot start, the device/system brings underlying activity to the foreground. If all the user application activities have remained in device memory, the app avoids having to "repeat object initialization, layout inflation," as well as rendering.
  • However, in situations where some memory had been purged while responding to "memory trimming events, such as onTrimMemory()," purged objects are recreated to implement a hot start event.
  • A hot start is known to display the same on-screen effect when compared to a cold start scenario. The system process usually displays a blank screen pending when the app renders the activity.
  • Whether some memory has been purged or not, the best practice published by within a developers' documentation guide on developing "games for all screens," recommends an interval less than two seconds as the KPI for a hot startup procedure.
  • A longer duration of a hot startup can lead to poor game startup experience, abandonment of an app, etc. Obviously, these negative events adversely affect bounce rates, lead to unfavorable return rates, and high uninstall rates.

ANDROID VITALS: PLAYSTORE BENCHMARKS

  • According to ProAndroidDev, a Play Store Android Vital better than" 85% of other apps" is excellent.
  • Achievable benchmarks are performance metrics found in Google Play Console and can be used as a practical monitoring tool.
  • Play Store Android Vitals has been in existence for over a year and a half now to help developers identify the problems worthy of solving. One among the most significant benefits of Play Sore Android Vitals is that they give developers metrics for free.
  • Developers gain access to Play Store Android Vitals by merely uploading their app to the Play Store. They also get more than ordinary numbers, as they can compare their app to other apps, allowing them to see how their apps are doing.
  • Android Vitals benchmarks are seen in the Google Play Console displayed "next to each metric."
  • According to ProAndroidDev, an app that is rated by Android Vitals among the "bottom 30% of startup times" is indicative of poor performance and underlying problems with the app.
  • A typical metric which reveals that an app is performing "better than 85% of other apps," is evidence of excellent performance and worthy of celebration but can serve as a motivation to get better performance.
  • Benchmarks can signal to developers if their app is performing better than other apps in terms of cold startups, etc.

CRASH RATE BENCHMARK

  • A potential KPI which can be used as the best practice metrics for cold, warm and hot startups of apps is the crash rate benchmark.
  • High crash rates, when used to monitor startups, might be a signification that several cold, warm, or hot startups failed to complete successfully. Statistics reveal that gaming apps with high crash rates are more prone to get "1-star reviews" from user ratings. Therefore, it is helpful to monitor crash rates and make relevant adjustments when necessary.
  • It is vital to monitor crash rates. Instances of app crashes allow Android to terminate an app's process while displaying a dialog to alert the user "that the app has stopped." This directly relates to higher bounce rates.

APP NOT RESPONDING (ANR) BENCHMARK

  • A potential KPI which can be used as the best practice metrics for cold startups, etc., of apps, is the ANR (App Not Responding) benchmark.
  • The ANR (App Not Responding) benchmark can also be used to monitor startups. Obviously, a high rate of ANRs (app not responding instances) indicates a corresponding high count of cold startup failures, etc.
  • According to Developer best practices, Android vitals reveals that ANRs are excessive when an app has at least "one ANR in at least 0.47%" among its daily sessions.
  • ANRs are also said to be excessive if two or more ANRs appear in at "least 0.24%" of an app's daily sessions.
  • Statistics reveal that high rates of ANRs are associated with 52% more instances of "1-star ratings" from user reviews. Therefore, it is helpful to monitor crash rates and make relevant adjustments when necessary.
  • It is vital to monitor ANR because ANR dialog gives app users an opportunity to quit the app forcefully leading to high bounce rates.

RESEARCH STRATEGY

Our team reviewed app developer publications like Android Developers, Overwolf Developers, etc., for potential KPIs and best practice metrics for cold, warm, and hot startups of apps (specific to gaming when possible). Android Developers in a documentation guide on developing "games for all screens," revealed some potential KPIs or best-practice metrics related to game performance. The documentation states that cold startup interval should be less than 5 seconds, warm startup interval (less than 2 seconds), and a hot startup interval (less than 1.5 seconds). ANRs are said to be excessive if two or more ANRs appear in at "least 0.24%" of an app's daily sessions.
Sources
Sources