Bluetooth Low Energy on Android, Part 3
AndroidIn this final installment, we will dive into the Client Characteristic Configuration Descriptor, which we’ll use to control notifications.
I Like Big Apps and I Cannot Lie: Using ProGuard to Avoid the Dalvik Method Limit
As the Android community grows, so does the vast offering of available third-party libraries. Utilizing them allows developers to quickly and easily expand the capabilities of an app. We love it, users love it, until suddenly: the mysterious compile error.
Unable to execute dex: method ID not in [0, 0xffff]: 65536
Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536
Or the new version:
trouble writing output:
Too many field references: 131000; max is 65536.
You may try using --multi-dex option.
The usual clean-and-build approach doesnt work, nor does restarting the ever-stable Android Studio. After a quick search, you will have probably found that you’ve hit quite the wall. You have encountered the infamous Dalvik method limit. As seen above, that limit is 65,536 methods. This count includes not just your code, but each third-party method you call upon.
So how do you know exactly what is eating into that number? One of my favorite tools is dex-method-counts. It runs on the APK from your application, so unfortunately you may need to undo the previous work that pushed you over the limit. Running it on my current project reports the number of methods in each package.
Processing app-dev-debug-0.1-debug.apk
Read in 63286 method IDs.
<root>: 63286
: 6
android: 17005
support: 13532
butterknife: 182
com: 33637
bignerdranch: 625
facebook: 3318
google: 12464
android: 10592
gms: 10592
gson: 964
maps: 402
jumio: 2421
mobileapptracker: 565
mobsandgeeks: 211
saripaar: 211
newrelic: 3100
myproject: 8315
squareup: 2198
okhttp: 1625
picasso: 536
dagger: 268
de: 186
greenrobot: 186
io: 569
branch: 569
java: 1562
javax: 134
jumiomobile: 3047
okio: 461
org: 5256
joda: 4711
json: 66
retrofit: 494
Overall method count: 63286
Before you try to count all of these, know that the output has been trimmed to just the higher levels in order to save space. However, you can still see we are using quite a few libraries and are about to go over the Dalvik limit. If development is to continue, what can we do?
ProGuard to the Rescue!
Short of removing libraries and their related features (lol) or combining methods (gross, no), we are left with ProGuard. Now you might say “Hey! What about the fancy new Multidex option?” You could use Multidex, but it’s more of a band-aid than a magic solution. While it has its own support library for pre-Lollipop Android, there are a few known issues. These can include applications not starting or responding, random out of memory crashes, or library incompatibilities with a secondary dex file. These all sound like an unpleasant experience for our users!
So what does ProGuard do exactly? The Android Developers page sums it up best:
The ProGuard tool shrinks, optimizes, and obfuscates your code by removing unused code and renaming classes, fields, and methods with semantically obscure names.
That sounds exactly like what we need; awesome! When using Gradle, we can turn it on by adding the example code into our build type in the app build.gradle:
minifyEnabled true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
Build and ship! …Right? Sadly, no.
Warning:butterknife.internal.ButterKnifeProcessor: can't find superclass or interface javax.annotation.processing.AbstractProcessor
Warning:retrofit.RxSupport$1: can't find superclass or interface rx.Observable$OnSubscribe
Warning:library class dagger.internal.codegen.GraphAnalysisErrorHandler extends or implements program class dagger.internal.Linker$ErrorHandler
Warning:library class dagger.internal.codegen.GraphAnalysisInjectBinding extends or implements program class dagger.internal.Binding
- trimmed 3 more similar lines -
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.annotation.processing.AbstractProcessor
- trimmed 34 more similar lines -
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced field 'javax.annotation.processing.ProcessingEnvironment processingEnv' in program class butterknife.internal.ButterKnifeProcessor
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.annotation.processing.ProcessingEnvironment
- trimmed 100 more similar lines -
Warning:com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl: can't find referenced method 'long getContentLengthLong()' in program class com.squareup.okhttp.internal.huc.HttpURLConnectionImpl
Warning:com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl: can't find referenced method 'long getHeaderFieldLong(java.lang.String,long)' in program class com.squareup.okhttp.internal.huc.HttpURLConnectionImpl
Warning:okio.DeflaterSink: can't find referenced class org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement
Warning:okio.Okio: can't find referenced class java.nio.file.Files
- trimmed 10 more similar lines -
Warning:retrofit.RestMethodInfo$RxSupport: can't find referenced class rx.Observable
- trimmed 15 more similar lines -
Warning:retrofit.appengine.UrlFetchClient: can't find referenced class com.google.appengine.api.urlfetch.HTTPMethod
- trimmed 36 more similar lines -
Warning:there were 266 unresolved references to classes or interfaces.
You may need to add missing library jars or update their versions.
If your code works fine without the missing classes, you can suppress
the warnings with '-dontwarn' options.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass)
Warning:there were 7 instances of library classes depending on program classes.
You must avoid such dependencies, since the program classes will
be processed, while the library classes will remain unchanged.
(http://proguard.sourceforge.net/manual/troubleshooting.html#dependency)
Warning:there were 3 unresolved references to program class members.
Your input classes appear to be inconsistent.
You may need to recompile the code.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedprogramclassmember)
:app:proguardDevDebug FAILED
Error:Execution failed for task ':app:proguardDevDebug'.
> java.io.IOException: Please correct the above warnings first.
Information:BUILD FAILED
Information:Total time: 20.137 secs
Information:1 error
Information:207 warnings
Now what do we do? The obvious answer is to crawl the web, but let’s see if we can decipher some of these messages. It would seem that we ProGuard’d a little too hard, and it has removed many files/methods that our app still needs. If you remember, we told ProGuard to use proguard-rules.pro. This file can be used to protect files, packages, fields, annotations and all sorts of things from being removed during the shrinking process. It can also be used to quiet warnings we know to be false.
We can see that there are a lot of problems related to Butter Knife, so let’s tackle that one first. We know that Butter Knife uses annotations, so we need to keep those.
-keepattributes *Annotation*
And we should probably keep the library intact by not removing anything that is in the butterknife package.
-keep class butterknife.** { *; }
If you have read up on how Butter Knife internals work or poked around in your output files, you would have found out that it generates classes that it accesses via reflection in order to connect your views. ProGuard cannot figure this out, so we must tell it to keep the classes named MyClass$$ViewInjector.
-keep class **$$ViewInjector { *; }
These next two are not as straightforward, but related to the previous reflection process. We need to keep all fields and methods that Butter Knife may use to access the generated injection class.
-keepclasseswithmembernames class * {
@butterknife.* <fields>;
}
-keepclasseswithmembernames class * {
@butterknife.* <methods>;
}
And finally, you will notice lots of repeated internal errors:
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.lang.model.type.DeclaredType
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.lang.model.element.TypeElement
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.lang.model.type.TypeMirror
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.lang.model.SourceVersion
Warning:butterknife.internal.ButterKnifeProcessor: can't find referenced class javax.lang.model.element.Element
According to ProGuard documentation:
If the missing class is referenced from a pre-compiled third-party library, and your original code runs fine without it, then the missing dependency doesn’t seem to hurt.
Unfortunately, this isn’t an answer with much logic. We now have two options: assume we need the missing class and keep it, or gamble. Since the goal is to remove unused code in order to pull us back from the Dalvik limit, I always attempt to ignore the warning first. This method requires thorough testing to be sure that nothing is broken and should be used with caution.
-dontwarn butterknife.internal.**
Building with these new rules will give us the following:
Warning:retrofit.RxSupport$1: can't find superclass or interface rx.Observable$OnSubscribe
Warning:library class dagger.internal.codegen.GraphAnalysisErrorHandler extends or implements program class dagger.internal.Linker$ErrorHandler
- trimmed 5 more similar lines -
Warning:com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl: can't find referenced method 'long getContentLengthLong()' in program class com.squareup.okhttp.internal.huc.HttpURLConnectionImpl
Warning:com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl: can't find referenced method 'long getHeaderFieldLong(java.lang.String,long)' in program class com.squareup.okhttp.internal.huc.HttpURLConnectionImpl
Warning:okio.DeflaterSink: can't find referenced class org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement
Warning:okio.Okio: can't find referenced class java.nio.file.Files
- trimmed 10 more similar lines -
Warning:retrofit.RestMethodInfo$RxSupport: can't find referenced class rx.Observable
- trimmed 15 more similar lines -
Warning:retrofit.appengine.UrlFetchClient: can't find referenced class com.google.appengine.api.urlfetch.HTTPMethod
- trimmed 36 more similar lines -
Warning:there were 93 unresolved references to classes or interfaces.
You may need to add missing library jars or update their versions.
If your code works fine without the missing classes, you can suppress
the warnings with '-dontwarn' options.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass)
Warning:there were 7 instances of library classes depending on program classes.
You must avoid such dependencies, since the program classes will
be processed, while the library classes will remain unchanged.
(http://proguard.sourceforge.net/manual/troubleshooting.html#dependency)
Warning:there were 2 unresolved references to program class members.
Your input classes appear to be inconsistent.
You may need to recompile the code.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedprogramclassmember)
:app:proguardDevDebug FAILED
Error:Execution failed for task ':app:proguardDevDebug'.
> java.io.IOException: Please correct the above warnings first.
Information:BUILD FAILED
Information:Total time: 24.352 secs
Information:1 error
Information:75 warnings
We’ve gone from 207 warnings down to 75; much better! But we still need to sort out the rest of the issues. Let’s try addressing the Retrofit issues, and start by keeping the library.
-keep class retrofit.** { *; }
Retrofit also uses annotated methods, but we have already added that rule for Butter Knife. But these methods could be removed by ProGuard because they have no obvious uses.
-keepclasseswithmembers class * {
@retrofit.http.* <methods>;
}
Retrofit utilizes Gson for serialization, and it needs the generic type information to be kept.
-keepattributes Signature
-keep class com.google.gson.** { *; }
Now add some rules for OkHttp.
-keep class com.squareup.okhttp.** { *; }
-keep interface com.squareup.okhttp.** { *; }
Much like Butter Knife, we have lots of duplicate warnings.
Warning:okio.Okio: can't find referenced class java.nio.file.OpenOption
Warning:okio.Okio: can't find referenced class org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement
Warning:retrofit.RxSupport$1: can't find referenced class rx.Observable$OnSubscribe
Warning:retrofit.RxSupport$2: can't find referenced class rx.Subscriber
Warning:retrofit.appengine.UrlFetchClient: can't find referenced class com.google.appengine.api.urlfetch.HTTPMethod
Warning:retrofit.appengine.UrlFetchClient: can't find referenced class com.google.appengine.api.urlfetch.URLFetchServiceFactory
These look to be internal or third party library warnings, so let’s try ignoring them.
-dontwarn com.squareup.okhttp.internal.huc.**
-dontwarn retrofit.appengine.UrlFetchClient
-dontwarn rx.**
-dontwarn okio.**
Let’s try running it now.
Warning:library class dagger.internal.codegen.GraphAnalysisErrorHandler extends or implements program class dagger.internal.Linker$ErrorHandler
- trimmed 5 more similar lines -
Warning:there were 7 instances of library classes depending on program classes.
You must avoid such dependencies, since the program classes will
be processed, while the library classes will remain unchanged.
(http://proguard.sourceforge.net/manual/troubleshooting.html#dependency)
:app:proguardDevDebug FAILED
Error:Execution failed for task ':app:proguardDevDebug'.
> java.io.IOException: Please correct the above warnings first.
Information:BUILD FAILED
Information:Total time: 7.491 secs
Information:1 error
Information:8 warnings
Only eight warnings left! Dagger seems to be the last library having problems. To resolve them, we’ll begin with adding the Dagger library and the methods it will use.
-keep class dagger.** { *; }
-keepclassmembers,allowobfuscation class * {
@javax.inject.* *;
@dagger.* *;
<init>();
}
And also the generated classes it uses for reflection.
-keep class **$$ModuleAdapter
-keep class **$$InjectAdapter
-keep class **$$StaticInjection
If you noticed above, we kept javax.inject.* related methods, but we also need the rest of that library.
-keep class javax.inject.** { *; }
And lastly let’s again quiet the internal library warnings:
Warning:library class dagger.internal.codegen.GraphAnalysisLoader extends or implements program class dagger.internal.Loader
Warning:library class dagger.internal.codegen.GraphAnalysisProcessor$1 extends or implements program class dagger.internal.BindingsGroup
With the same ignore rule:
-dontwarn dagger.internal.codegen.**
Let’s try to run it and see. No warnings! 🙂 But it crashes immediately 🙁 Get used to seeing this with ProGuard when adding new libraries. Can you guess what we forgot? All the other libraries! We have Event Bus, New Relic, Jumio and Saripaar. Event bus will need to keep its onEvent and onEventMainThread methods, so let’s add a rule for that.
-keepclassmembers, includedescriptorclasses class ** {
public void onEvent*(**);
}
-keepclassmembers, includedescriptorclasses class ** {
public void onEventMainThread*(**);
}
New Relic (3100 methods) may contribute a noticible bump towards the Dalvik limit, but I am inclined to leave it alone since it is our app monitoring framework. It has (and will need) its own exceptions and inner classes in order to correctly report on our app. Not to mention that it can help debug issues that we might introduce via ProGuard.
-keep class com.newrelic.** { *; }
-dontwarn com.newrelic.**
-keepattributes Exceptions, InnerClasses
Jumio (2421 methods) isn’t all that big and Saripaar (211) is almost negligible, so let’s just keep all of them.
-keep class com.jumio.** { *; }
-keep class com.mobsandgeeks.saripaar.** {*;}
-keep class commons.validator.routines.** {*;}
Try running again… and still a crash?! What went wrong now—haven’t we accounted for everything? After more research, I found that Dagger 1.2.2 has issues with obfuscation. Which is fine, since our main goal was to shrink. Let’s turn off that step and see what happens.
-dontobfuscate
Success! The app successfully builds and launches. Now for the real test, checking the method count to see how many we removed.
Processing app-dev-debug-0.1-debug.apk
Read in 44413 method IDs.
<root>: 44413
: 3
android: 10379
support: 7886
butterknife: 182
com: 25103
bignerdranch: 485
facebook: 1283
google: 7117
android: 5969
gms: 5969
gson: 964
maps: 46
jumio: 2421
mobileapptracker: 256
mobsandgeeks: 211
saripaar: 211
newrelic: 3100
myproject: 7882
squareup: 2071
okhttp: 1625
picasso: 412
dagger: 268
de: 66
greenrobot: 66
io: 444
branch: 444
java: 1378
javax: 134
jumiomobile: 3047
okio: 341
org: 2474
joda: 1995
json: 59
retrofit: 494
Overall method count: 44413
We started with 63,286 methods and now have 44,413. Thats almost 20,000 methods removed! Upon closer inspection, some of the greatest savings were from the biggest libraries. You may have noticed that we didn’t account for them, either. Thankfully, Google and Facebook provide their own ProGuard rules internally, saving us a lot of work.
Our own myproject and bignerdranch packages were trimmed up quite a bit, which is great. If we wanted or needed to remove more methods, we could attempt to shrink a bit more aggressively. Instead of using entire package rules like -keep class com.library.** { *; } we could dig deeper and understand exactly which parts our app needs.
Regardless of what you choose to do next, these are great results for a first pass. As you may have seen, it took only 10-20 seconds to use ProGuard on an app that wouldn’t finish building. Ideally, you would turn on ProGuard only for release builds, but you may have to enable it for debugging as you add more libraries.
Of course, we need to thoroughly test the app to make sure there are no problems with the shrinking process, but we have freed up a lot of space. We can now add more libraries to make our app even better. Just remember to add the appropriate ProGuard rules when you do!
Full dex-method-counts and build outputs, as well as the final set of ProGuard rules can be found in this gist.
