Shunix's Weblog

Analyzing Multidex Source Code

Shunix -

Recently I read the source code of Google's multidex library due to some reasons. I'll focus on how to load multiple dex files rather than how to split classes into multiple dex files in this post.

Entry Point

Integrated with multidex library is pretty easy. According to the official documentation, in order to use the multidex library, the application class should be MultiDexApplication or a descendant class of it. Obviously, we should start with this class. This class is really simple, it just calls a static method of MultiDex.

public class MultiDexApplication extends Application {
  @Override
  protected void attachBaseContext(Context base) {
    super.attachBaseContext(base);
    MultiDex.install(this);
  }
}

Let's continue with the install method of MultiDex. Due to the space limitation, I omitted some unimportant code like logs, variable checks, etc.

public static void install(Context context) {
	ApplicationInfo applicationInfo = getApplicationInfo(context);
	synchronized (installedApk) {
		String apkPath = applicationInfo.sourceDir;
		if (installedApk.contains(apkPath)) {
			return;
		}
		installedApk.add(apkPath);
		ClassLoader loader;
		loader = context.getClassLoader();
		clearOldDexDir(context);
		File dexDir = getDexDir(context, applicationInfo);
		List<File> files = MultiDexExtractor.load(context, applicationInfo, dexDir, false);
		if (checkValidZipFiles(files)) {
			installSecondaryDexes(loader, dexDir, files);
		} else {
			files = MultiDexExtractor.load(context, applicationInfo, dexDir, true);

			if (checkValidZipFiles(files)) {
				installSecondaryDexes(loader, dexDir, files);
			} else {
				throw new RuntimeException("Zip files were not valid.");
			}
		}
	}
}

This method do the following steps:

  • Check if the apk has already been processed
  • Clean old files in directory "/data/data/< package name >/files/secondary-dexes". This operation is for capability.
  • Create directory "/data/data/< package name >/code_cache/secondary-dexes" (getDexDir())
  • Extract dex files from apk by calling MultiDexExtrator.load() method
  • Check if the extraction succeed
  • Install additional dex files

I skipped some simple methods here like clearOldDexDir and getDexDir. It's very easy to understand the logic in these methods. We need to get clear about the extraction and installation process.

Extraction

Here's the code of MultiDexExtrator.load()

static List<File> load(Context context, ApplicationInfo applicationInfo, File dexDir,
            boolean forceReload) throws IOException {
	final File sourceApk = new File(applicationInfo.sourceDir);
	long currentCrc = getZipCrc(sourceApk);
	List<File> files;
	if (!forceReload && !isModified(context, sourceApk, currentCrc)) {
		try {
			files = loadExistingExtractions(context, sourceApk, dexDir);
		} catch (IOException ioe) {
			putStoredApkInfo(context, getTimeStamp(sourceApk), currentCrc, files.size() + 1);

		}
	} else {
		files = performExtractions(sourceApk, dexDir);
		putStoredApkInfo(context, getTimeStamp(sourceApk), currentCrc, files.size() + 1);
	}
	return files;
}

This method check the crc of base apk first. The isModified method checks both the time stamp and crc32 of base apk, if both are the same and forceReload is false, then the existing extractions will be used, loadExistingExtractions method just do the validation. If isModified returned true or forceReload is true, extraction will be performed. After extraction, time stamp and crc will be stored into shared preference. Here's the code of performExtractions

private static List<File> performExtractions(File sourceApk, File dexDir)
            throws IOException {
	final String extractedFilePrefix = sourceApk.getName() + EXTRACTED_NAME_EXT;
	prepareDexDir(dexDir, extractedFilePrefix);
	List<File> files = new ArrayList<File>();
	final ZipFile apk = new ZipFile(sourceApk);
	try {
		int secondaryNumber = 2;
		ZipEntry dexFile = apk.getEntry(DEX_PREFIX + secondaryNumber + DEX_SUFFIX);
		while (dexFile != null) {
			String fileName = extractedFilePrefix + secondaryNumber + EXTRACTED_SUFFIX;
			File extractedFile = new File(dexDir, fileName);
			files.add(extractedFile);
			int numAttempts = 0;
			boolean isExtractionSuccessful = false;
			while (numAttempts < MAX_EXTRACT_ATTEMPTS && !isExtractionSuccessful) {
				numAttempts++;
				extract(apk, dexFile, extractedFile, extractedFilePrefix);
				isExtractionSuccessful = verifyZipFile(extractedFile);
				if (!isExtractionSuccessful) {
					extractedFile.delete();
					if (extractedFile.exists()) {
					}
				}
			}
			if (!isExtractionSuccessful) {
			}
			secondaryNumber++;
			dexFile = apk.getEntry(DEX_PREFIX + secondaryNumber + DEX_SUFFIX);
		}
	} finally {
		try {
			apk.close();
		} catch (IOException e) {
		}
	}
	return files;
}

The extractedFilePrefix is "base.apk.classes". The prepareDexDir method will remove all the files in dexDir("/data/data/< package name >/code_cache/secondary-dexes") whose file name do not started with base.apk.classes. Dex files in apk file is named like "classes1.dex", "classes2.dex" and so on. All these dex files will be extracted and re-zipped into "base.apk.classes.zip" Those zip files only contains one entry named classes.dex per file, they're stored in dexDir("/data/data/< package name >/code_cache/secondary-dexes"). If the extraction succeed, a verification is performed to ensure the compression is successful.
Here we're done with the extraction.

Installation

As we have finished the extraction, here comes the problem, how do we make the application load classes from extracted dex file? We could get the classloader from the context, and this classloader should be a descendant of dalvik.system.BaseDexClassLoader. We just need to modify the field pathList of it to append additional dex files. The loading process of dex files is complicated, I'll talk about it on subsequent post. Let's have a look at the installSecondaryDexes method.

private static void installSecondaryDexes(ClassLoader loader, File dexDir, List<File> files)
            throws IllegalArgumentException, IllegalAccessException, NoSuchFieldException,
            InvocationTargetException, NoSuchMethodException, IOException {
        if (!files.isEmpty()) {
            if (Build.VERSION.SDK_INT >= 19) {
                V19.install(loader, files, dexDir);
            } else if (Build.VERSION.SDK_INT >= 14) {
                V14.install(loader, files, dexDir);
            } else {
                V4.install(loader, files);
            }
        }
    }

There're some slight differences between different Android versions. V14.install is the most typical one, so let's take it as example.

private static final class V14 {

    private static void install(ClassLoader loader, List<File> additionalClassPathEntries,
            File optimizedDirectory)
                    throws IllegalArgumentException, IllegalAccessException,
                    NoSuchFieldException, InvocationTargetException, NoSuchMethodException {
        Field pathListField = findField(loader, "pathList");
        Object dexPathList = pathListField.get(loader);
        expandFieldArray(dexPathList, "dexElements", makeDexElements(dexPathList,
                new ArrayList<File>(additionalClassPathEntries), optimizedDirectory));
    }

    private static Object[] makeDexElements(
            Object dexPathList, ArrayList<File> files, File optimizedDirectory)
                    throws IllegalAccessException, InvocationTargetException,
                    NoSuchMethodException {
        Method makeDexElements =
                findMethod(dexPathList, "makeDexElements", ArrayList.class, File.class);

        return (Object[]) makeDexElements.invoke(dexPathList, files, optimizedDirectory);
    }
}

Reflection is used here. The install method take the following steps:

  • get the instance of "pathList" from classloader
  • find the array "dexElements" from "pathList" instance
  • combine the original array and addition dex file entries, then set the value back to "dexElements"

The V19.install method is almost the same with V14.install except that the makeDexElements method signature is slightly different. We have to pass an array list of IOException as the last parameter when calling makeDexElements. After the method returned, we need to handle this array list like DexPathList did.

It's different on Android V4-V13. The classloader is a descendant of dalvik.system.DexClassLoader, so we need to change the code accordingly. In brief, we modified the mPaths, mFiles, mZips and mDexs to append additional dex files. As Android V4-V13 is not the mainstream and the code is easy to understand if we've known the logic, I don't paste the code here.

The installation of additional dex files is achieved in this way. Now the application can access the classes in additional dex files just like they're in the main dex. From the code, we can know that there're still some problems with multidex.

  1. As the MultiDex.install() is executed in main thread, and there're I/O operations in it, if the additional dex files are large, it may cause the ANR.
  2. Due to the LinearAlloc bug, LinearAlloc size is 5M on Gingerbread and below. With multidex library, the installation of apk may succeed, while the application may fail to start on these devices.

There's still room for improvement when handling additional dex files.