How to build AOSP with Docker

I recently had the need to download and compile the Android Open Source Project (AOSP). Although I knew that it would take more than a day to set up my environment, download the source, compile the system and then test it, I had no idea that it would actually take me over three weeks to get everything working.

I run Linux (Fedora 20) on my computer and figured that it should be pretty straightforward to set up everything. I started with the official documentation on how to build Android. And then searched for some added documentation on customizing the environment for Fedora. This took some time because even the blog that was specialized for Fedora 20 had some incongruities with my system.

After finally getting my environment set up it was time to download the code. You can find the tags that you need here. Downloading was pretty straightforward and without problems but took over 2 hours.

Finally I was all set up and ready to compile. This did not go well, at all. Many of the applications and tools that were installed did not work as expected. The build was stopped immediately because I did not have the correct version of Java. This was very annoying to me because I had actually changed my version of Java to match what the documentation asked for.

Java version error when compiling AOSP

So I removed OpenJDK and installed Oracle 1.7. And this also failed as it required 1.6. (I didn't take a screen shot of that error, sorry.)  I had to install-uninstall multiple versions of Java on my computer, surely this was not good for my computer in the long run.

Once I finally had the Java installation in a stable condition it was time to try compiling again. And this went mostly well until I got a compile error stating that a file was corrupted. This was a difficult problem which ultimately led to a StackOverflow question. I recommend that you read the answer to that question as it may affect your every day usage of Linux. Once that was fixed I started the build again. But it would often break and stop, I would just restart it and it would go further, break and repeat. It wasn't very confidence inspiring to say the least.

One error that I did get that required a fix was this one:

 make: *** No rule to make target `frameworks/rs/scriptc/rs_core_math.rsh', needed by `out/target/common/obj/APPS/Galaxy4_intermediates/src/RenderScript.stamp'. Stop.  

This was pretty much the only error that had results from a Google search. The answer was to copy the source code from the Android source here. This did work but I don't really like this solution as this file is supposed to be generated and it wasn't.

Finally (day have passed by now), the source code completely compiled! It was time to launch it in the emulator (which was/is my goal, I have no need for this to run on a device). I launched the 'emulator' command and after a long while the emulator showed my home screen. Success!

Time to start up Android Studio and install my application! Which, of course, didn't work. After a few quick tries with AS I wanted to cut out the middle man and went straight to the command line. It tried 'adb shell pm install' which also failed. 'pm' is actually a script, so I went to the main executable 'app_process' which was the culprit and that resulted in this StackOverflow question about a segmentation fault (which didn't get answered).  I then tried a new approach, I would bake my app right into the ROM. There are a few approaches to this, one of which is adding your code to the AOSP code base and compile the whole thing in one go. I wanted something simpler, just adding my apk to the build. I was finally able to do that and see my application run in the ROM.
At this point, things were looking up, and yet bleak at the same time, I was getting closer to my goal, but I was plague with problems and I didn't trust my build at all. There must be a better way. This is where Docker comes in.

About Docker

Someone commented on one of my Google+ posts saying that there was a Docker container for AOSP builds. That comment stayed on the back burner for a few days. Now was the time to check it out. I read up on Docker, watched a video and basically came up to speed. I was excited by this new idea which was much more portable and lean than a Virtual Machine. My first step was to find this Docker container that this person had mentioned. I searched on the Docker hub and although there were some containers there they did not work for me, and they had little to no documentation.

More searching brought me to some GitHub projects. I didn't really understand what was going on until I researched Dockerfiles. This was the ticket. I would get one of the docker files on GitHub, build it and my troubles would be over. Which of course didn't work. So I built my own Dockerfile, you will learn how to use it in the next section.

Installing Docker

Docker can be installed by many package managers on Linux.It can also be installed on other operating systems such as Mac or Windows. I installed it by following the instructions for Fedora 20. I will let you find the correct installation procedure for your OS by clicking one of those links and choosing it from the 'Installation' dropdown.
Installation was, of course, a breeze. However, even after adding my username to the 'docker' group, I still have to run Docker with sudo. I will try to fix this in the future but for now it's not my main focus.

Setting up your environment

This is where Docker shines! Usually you would go to the Initializing a Build Environment page and hope that you are on the correct version of Ubuntu and that you can get all the correct versions of all the tools. Not today!

First, let's make sure that your Docker daemon is running.

 $ sudo systemctl start docker.service  

With your Docker installation and the Dockerfile that I am providing this will all be done in one line. Feel free to go view the contents of the Dockerfile here.

 $ sudo docker build -t <repositoryName/aosp> https://github.com/yartiss/docker-aosp-build.git

where you replace 'repositoryName' with your user name. It doesn't matter what name you use for now.

Running this command will take a while. But once it is done you will have a complete environment (called an image in Docker speak) in which you can compile AOSP!

We can then create a container by using the 'docker run' command on this image. (Note: This will not be the actual command that you use to create your real compilation container. This is just to see your container run)

 $ sudo docker run -i -t <repositoryName>/aosp /bin/bash  

Here we are telling docker to run a new container based on the <repositoryName>/aosp image and to launch 'bash' within that container. The '-i' argument tells Docker that we want an interactive container so that we can input commands. You should now see something like this:

 builder@bc338942ef20:/$  

Where 'builder' is the account's user name and 'bc338942ef20' is the ID of the container. The ID will always be recreated when you create a new container. You are now inside of your Ubuntu container! You can use it just like a normal Ubuntu system. You will find that all of the tools that you need to build AOSP are there, ready and waiting for you. You can try a few commands inside the container to take it for a test spin, and then close the container and return to your host OS by executing "exit".

Downloading AOSP

This section is a bit special because I had already downloaded the AOSP source code before deciding on using Docker. This means two things:

1) I would have to get the source code inside the docker container

2) I would have to get the ROM back out in order to test it.

I have solve these two problems by using a Docker Data Volume. A data volume is basically a reference to a file or folder on your host machine. Therefore the container can use the files from the host file system and changes to the files made within the container are seen by the host file system. Simply put the Docker container will be able to use my existing AOSP source code, and the ROM that is built inside the container will be accessible from my Fedora as usual.

What this means for you is that you will now download the AOSP source code onto your host operating system (not the container), if you don't already have it. To do this you will follow the official download documentation. Note down the AOSP root directory as you will need it in the next step.

Building the source code

Launching the build container

Now that you have Docker set up and the AOSP source code, it's time to build it.

1) Start a new terminal instance from your OS

2) Start the Docker daemon in that terminal

 $sudo systemctl start docker.service  

3) Run a new build container.

 sudo docker run -i -t -v </path/to/Android_source>:/home/builder/Android_source -v /etc/localtime:/etc/localtime:ro <repositoryName>/aosp /bin/bash  

where </path/to/Android_source> is the root directory of the AOSP source code on your host OS and where <repositoryName> is the name with which you created the docker image.

This command will start up your build environment inside your Ubuntu container and wire up the data volume to your AOSP source code on your host machine. There is also a parameter that will set your container's time to your host's time.

The Build!

You are now ready to build your ROM. This is done by following the normal build procedure.

 $ . build/envsetup.sh  
 $ lunch aosp_x86-eng 
 $ make -j4  

That's it. Your build should start and finish in a few hours. See below if you have any problems

Good luck!

Note: This tutorial will create a ROM that is only meant to run in the emulator, this is why you should select the "aosp_x86-eng" build configuration.

The problems

SELinux

In my case, I have SELinux running on my host operating system. This caused an error or two when the container tried to access some files from my host OS. The first problem was that I was not able to 'cd' into the AOSP directory from within my container. And the second problem looked like this:

 error while loading shared libraries: /usr/lib/libavformat.so.50: cannot restore segment prot after reloc: Permission denied  

Starting up the 'SELinux Troubleshooter' application showed me the errors and the exact command to apply.

 If you believe that dexopt should be allowed execmod access on the file by default.  
 Then you should report this as a bug.  
 You can generate a local policy module to allow this access.  
 Do  
 allow this access for now by executing:  
 $ sudo grep dexopt /var/log/audit/audit.log | sudo audit2allow -M mypol  
 S sudo semodule -i mypol.pp  

Proguard

I also had problems with Proguard not being the correct version. After a web search I was able to cherry-pick the Proguard source code. I had to admit that this was really weird because I have two version of Proguard in my source tree, I don't know if this is normal. The fix was to cherry pick the following commit 8a6199f0c36a778f22394364347a301b0b28e94b into the <source_root>/external/proguard/ project.

 <source_root>/external/proguard $ git cherry-pick 8a6199f0c36a778f22394364347a301b0b28e94b  

Running your ROM in the emulator

Once you have a successful build you will want to see your baby run.  You will want to do this from your host environment, not your build environment. You can do this be running the 'exit' command from inside your container to get back to you host environment, but most likely you will just start up another terminal beside this one. Once inside this new terminal you will need to run the environment set up commands again so that your environment variables will be initialized.

  $ . build/envsetup.sh   
  $ lunch aosp_x86-eng  

Notice that we do not run the 'make' command. Now that our environment is set up we are one command away from seeing the results of our labour.

  $ emulator  

After a little while you should see the Android emulator running your ROM. Bravo!

Running a more useful version of your ROM

Now that your ROM is running in the emulator you will probably need a more useful version of your ROM.  There are many options that you can add to the emulator. You will probably want an SD Card and a camera. Let's add those right now. As a reminder, all of these steps are done in your host operating system, not inside the Docker container.

The SD Card

In order for the emulator to present you with an SD Card when using your ROM you must first create an SD Card image and then pass that image as an argument to the emulator. You create an SD Card image by using the mksdcard command.

  $ mksdcard -l mySdCard 1024M mySdCardFile.img  

Now call the emulator with the correct parameters in order to test the SD Card.

  $ emulator -sdcard <filepath>/mySdCardFile.img  

The camera

In order to use a camera you have a few choices. You can emulate a camera or you can use a webcam as a camera. In order to use a webcam for a camera you must first list the webcams that are available on your computer.

  $ emulator -webcam-list  
 emulator: WARNING: system partition size adjusted to match image file (550 MB > 200 MB)  
 List of web cameras connected to the computer:  
  Camera 'webcam0' is connected to device '/dev/video0' on channel 0 using pixel format 'YUYV'  

Now we can call the emulator with the correct parameters enabling you to use camera applications.

  $ emulator -camera-front webcam0  

bringing it all together

Now that we have out options set up, let's call the emulator with all of them

  $ emulator -sdcard <filepath>/mySdCardFile.img -camera-front webcam0  

Conclusion

For most cases it should have taken you a day from reading this post to running your own ROM in your emulator. It took me about a month the first time I tried. Docker is a pretty amazing product and I hope that this post has made you want to find out more about it and about creating your own ROM.

Building multiple APKs inside Android Studio

Build variants are a huge feature for Android development. The use cases for such a feature are many. It can be as simple as having a free version of your app and a paid version. Or maybe you are making generic software that can be sold to many companies that all want they're own branding.

Whatever your use case may be, Gradle comes to the rescue. Not only can you configure your variants but you can also select which ones actually get built. You are probably already aware of the standard way to do this in Android Studio which is to use the "Build Variants" tool window and select the variant that you want to build.

This builds one variant at a time and is very practical while you are developing as it allows you to quickly switch from one variant to the another.

Once development is done, you will want to build all of your release variants for final testing and distribution. You can do this in Android Studio as well. Simply open the "Gradle Tasks" tool window, which is usually on the right. You will see many tasks that start with 'assemble', double click on one of those and your APKs will be created.

For example, double clicking on 'assembleRelease' will create all your release apks.

From the docs:

Building and Tasks

We previously saw that each Build Type creates its own assemble task, but that Build Variants are a combination of Build Type and Product Flavor.

When Product Flavors are used, more assemble-type tasks are created. These are:

1) assemble[Variant Name]

2) assemble[Build Type Name]

3) assemble[Product Flavor Name]

1) allows directly building a single variant. For instance assembleFlavor1Debug.

2) allows building all APKs for a given Build Type. For instance assembleDebug will build both Flavor1Debug and Flavor2Debug variants.

3) allows building all APKs for a given flavor. For instance assembleFlavor1 will build both Flavor1Debug and Flavor1Release variants.

The task assemble will build all possible variants.

Note the "Recent tasks" section which allows you to quickly execute previously run tasks.

You'll also note that the tasks that you run are added to your Configurations dropdown. This allows you to quickly access them by hitting Shift-F10.

And that's it! I hope that this helps you out.

Localized getString() with parameters

getString

We all know the getString() method which is available in the Resources class and in the Context class. However, did you know that there is also a second version of that method that accepts parameters that will be merged into a format string? You can find the Android documentation here.

Simple example

Here is an example of this version of getString():

In your xml file:

 <string name="hello_person">Hello %1$s, how are you?</string>  

In your Java file:

 String userName = getUserName();  
 String hello = getResources().getString(R.string.hello_person, userName);  

This simplifies your code a lot. And it really is but a convenience at you can see in the source Android code:

 public String getString(int id, Object... formatArgs) throws NotFoundException {  
   String raw = getString(id);  
   return String.format(mConfiguration.locale, raw, formatArgs);  
 }  

Multipe placeholders example

You can also have more than one placeholder in your original string, and you can use more than one type:

In your xml file:

 <string name="hello_person_age">Hello %1$s, you are %2$d years old</string>   

In your Jave file:

 String userName = getUserName();  
 int age = getAge();  
 String hello = getResources().getString(R.string.hello_person_age, userName, age);   

Context class

And how about the version in the Context class?  It just delegates to the method in the Resources class:

 public final String getString(int resId, Object... formatArgs) {  
   return getResources().getString(resId, formatArgs);  
 }  

Thanks to +Wolfram Rittmeyer for tuning me into this version of getString()!

Automating ADB over Wi-Fi for multiple devices

Here is a script that will scan through your Android devices that are connected to ADB via a USB cable and then connect them all via ADB over Wi-Fi.

Usage is very simple.  If needed, modify the script with your IP range and then run the script. You will need to run this script every time you want to connect to your Wi-Fi network.

Note that I haven't yet found a way to test if a device is already connected via Wi-Fi so you may get errors such as "unable to connect to 192.168.1.2:5555:5555".  You'll know that the device is already connected if you see the double port value at the end.

Thanks to +Tony Owen for the original idea and to +Wolfram Rittmeyer with the help on the script.

Please feel free to expand this script or modify it. Let me know if you do so that I can update this post!

 #!/bin/bash  
   
 #Modify this with your IP range  
 MY_IP_RANGE="192\.168\.1"  
   
 #You usually wouldn't have to modify this  
 PORT_BASE=5555  
   
 #List the devices on the screen for your viewing pleasure  
 adb devices  
 echo   
   
 #Find USB devices only (no emulators, genymotion or connected devices  
 declare -a deviceArray=(`adb devices -l | grep -v emulator | grep -v vbox | grep -v "${MY_IP_RANGE}" | grep " device " | awk '{print $1}'`)  
   
 echo "found ${#deviceArray[@]} device(s)"  
 echo   
   
 for index in ${!deviceArray[*]}  
 do  
   echo "finding IP address for device ${deviceArray[index]}"  
   IP_ADDRESS=$(adb -s ${deviceArray[index]} shell ifconfig wlan0 | awk '{print $3}')   
     
   echo "IP address found : $IP_ADDRESS "  
     
   echo "Connecting..."  
   adb -s ${deviceArray[index]} tcpip $(($PORT_BASE + $index))  
   adb -s ${deviceArray[index]} connect "$IP_ADDRESS:$(($PORT_BASE + $index))"  
   
   echo  
   echo  
 done  
   
 adb devices -l  
 #exit  
   

Copying a template project

Today I fell upon a problem that really baffled me for a while.  Android Studio just refused to create my R class.  No lint problems, no compile problems (besides the missing R), nothing.  It just failed. I searched on the net but problems like that are difficult to find answers to because Android Studio is in such fluctuation that the answers no longer apply.

I tried cleaning my project. Rebuilding my project. It was then suggested that I invalidate my caches and restart Android Studio (you can find that in the File menu) but that didn't work.  I deleted my build directory. Nothing would work. Then it dawned on me...

I was trying to create a project from a template project that I created.  The Coloroos project will have many coloring books and each will require it's own project, so I wanted to create a template project and just copy paste it and go. What I did was copy and paste the project and then open it in Android Studio. There lies the problem. In order to properly reset everything I need to delete the .idea directory from the pasted project and then import it into Android Studio. Once I did that, it compiled and ran immediately.

Creating distinct PendingIntents

With my recent work on the BiPolar widget I came upon an interesting bug in my code.  One that made me scratch my head while walking through the code.

The widget is rather simple, it displays the current temperature in Celsius and Fahrenheit simultaneously and allows you to click on it if you want to share the current temperature with your favourite social media site. Everything seemed to be working fine until I tried installing multiple instances of the widget.  At the point, clicking on any widget would always share the weather of the last installed widget.  I figured that I had a logic problem in my code and poured over it repeatedly.  Nothing.

Here is the original erroneous code :

// Create the Intent that will be sent when the widget is clicked on
Intent intent = new Intent(this, BiPolar.class);
intent.setAction(CLICK_ACTION);
intent.putExtra(CLICK_WIDGET_ID, appWidgetId);

// Create the PendingIntent for the RemoteViews
PendingIntent pendingIntent = PendingIntent.getBroadcast(context, 0, intent, 0);

//attach an on-click listener to the widget
views.setOnClickPendingIntent(R.id.root, pendingIntent);

That code is inside of a loop that goes through all of the appWidgetIds and appears correct.  The problem lies in the fact that PendingIntent are reused by the system.  If the PendingIntent that would be returned by getBroadcast() or getActivity() is considered equal to one that already exists then a new one is not created, rather a copy of an existing one is returned. The problem lies in that Extras are not considered in the equality comparison. So in this case every call for a new PendingIntent returned the same PendingIntent over and over again.

From the documentation : 

A PendingIntent itself is simply a reference to a token maintained by the system describing the original data used to retrieve it. This means that, even if its owning application's process is killed, the PendingIntent itself will remain usable from other processes that have been given it. If the creating application later re-retrieves the same kind of PendingIntent (same operation, same Intent action, data, categories, and components, and same flags), it will receive a PendingIntent representing the same token if that is still valid, and can thus call cancel() to remove it.

Because of this behavior, it is important to know when two Intents are considered to be the same for purposes of retrieving a PendingIntent. A common mistake people make is to create multiple PendingIntent objects with Intents that only vary in their "extra" contents, expecting to get a different PendingIntent each time. This does not happen. The parts of the Intent that are used for matching are the same ones defined by Intent.filterEquals. If you use two Intent objects that are equivalent as per Intent.filterEquals, then you will get the same PendingIntent for both of them.

The Solution

So how to we make sure that the requested PendingIntent is different from the existing ones?  The solution is rather easy.  Again from the documentation : 

you will need to ensure there is something that is different about them to associate them with different PendingIntents. This may be any of the Intent attributes considered by Intent.filterEquals, or different request code integers supplied to getActivity(Context, int, Intent, int), getActivities(Context, int, Intent[], int), getBroadcast(Context, int, Intent, int), or getService(Context, int, Intent, int).

In my case the fix was very easy, just add the appWidgetId as the request code to the parameters of getBroadcast.

// Create the PendingIntent for the RemoteViews
PendingIntent pendingIntent = PendingIntent.getBroadcast(context, appWidgetId, intent, FLAG_UPDATE_CURRENT);

Although that field is not yet used, and may never be, it is enough for the system to distinguish the PendingIntent as being different from the other and to create a new one.