[java-source-utils] Add Java source code utilities (#623)

Context: https://github.com/javaparser/javaparser/tree/javaparser-parent-3.16.1
Context: #623

There are two parts of the current `.jar` binding toolchain which are
painful and could be improved:

 1. Parameter names
 2. Documentation extraction

Parameter names (1) are important because they become the names of
event members as part of ["event-ification"][0].  As such they are
semantically important, and the default behavior of "p0" makes for a
terrible user experience.

*If* the `.class` files in the `.jar` file are built with
`javac -parameters` (4273e5c), then the `.class` file will contain
parameter names and we're good.  However, this may not be the case.

If the `.class` files are built with `javac -g`, then we'll try to
deduce parameter names from debug info, but that's also problematic.

What else can be used to provide parameter names?

It is not unusual for Java libraries to provide "source `.jar`" files,
e.g. Android provides `android-stubs-src.jar` files, and other
libraries may provide a `*-sources.jar` file.  The contents of these
files are *Java source code*.  These files are used by Android IDEs to
provide documentation for the Java library.  They contain classes,
methods, parameter names, and associated Javadoc documentation.

What they are *not* guaranteed to do is *compile*.  As such, we can't
compile them ourselves with `javac -parameters` and then process the
`.class` files, as they may refer to unresolvable types.

"Interestingly", we *already* have some tooling to deal with this:
`tools/param-name-importer` uses a custom Irony grammar to parse
the Android SDK `*-stubs-src.jar` files to grab parameter names.
However, this tooling is *too strict*; try to pass arbitrary Java
source code at it, and it quickly fails.

Which brings us to documentation (2): we have a [javadoc2mdoc][1] tool
which will parse Javadoc HTML documentation and convert it into
[**mdoc**(5)][2] documentation, which can be later turned into
[XML documentation comments][3] files by way of
[**mdoc export-msxdoc**(1)][4], but this tool is (1) painful to
maintain, because it processes Javadoc *HTML*, and
(2) *requires Javadoc HTML*.

Google hasn't updated their downloadable Javadoc `.zip` file since
API-24 (2016-October).  API-30 is currently stable.

If we want newer docs, we either need to scrape the
developer.android.com/reference website to use with the existing
tooling, or...  we need to be able to read the Javadoc comments within
the `*-stubs-src.jar` files provided with the Android SDK.
(Note: Android SDK docs are Apache 2; file format conversion is fine.)

We thus have two use-cases for which parsing Java source code would
be useful.

As luck would have it, there's a decent Apache 2-licensed Java project
which supports parsing Java source code: [JavaParser][5].

Add a new `tools/java-source-utils` program which will parse Java
source code to produce two artifacts: parameter names and
consolidated Javadoc documentation:

	$ java -jar java-source-utils.jar --help
	java-source-utils [-v] [<-a|--aar> AAR]* [<-j|--jar> JAR]* [<-s|--source> DIRS]*
		[--bootclasspath CLASSPATH]
		[<-P|--output-params> OUT.params.txt] [<-D|--output-javadoc> OUT.xml] FILES

Provide `--output-params OUT.params.txt`, and the specified file will
be created which follows the file format laid out in
[`JavaParameterNamesLoader.cs`][6]:

	package java.lang
	;---------------------------------------
	  class Object
	    wait(long timeout)

Provide `--output-javadoc api-javadocs.xml`, and the resulting file
will be a `class-parse`-like XML file which uses `//@jni-signature` as
the "key" and a child `` element to contain docs, e.g.:

	
	  
	    
	      …
	      
	        …
	      
	      
	        
	        …
	      
	    
	  

This should make it possible to update the Xamarin.Android API
documentation without resorting to web scraping (and updating the code
to deal with whatever new HTML dialects are now used).

If neither `--output-params` nor `--output-javadocs` is used, then
`--output-javadocs` will be executed, writing to stdout.

The XML file *also* contains parameter name information, so that one
file can be the "source of truth" for parameter names and
documentation.

`FILES` can be:

  * Java source code in a `.java` file; or
  * A file with a `.jar` or `.zip` extension, which will be extracted
    into a temp directory and all `.java` files within the directory
    will be processed; or
  * A directory tree, and all `.java` files will be processed.

If a single file references other types, the "root" directory containing
those types may need to be specified via `--source DIR`:

	$ java -jar "bin/Debug/java-source-utils.jar" -v \
	  -s $HOMEandroid-toolchain/sdk/platforms/_t  \
	  $HOME/android-toolchain/sdk/platforms/_t/android/app/Activity.java \
	  -P android.params.txt -D android-javadocs.xml >o.txt 2>&1

TODO:

In some scenarios, types won't be resolvable.  What should output be?

We don't want to *require* that everything be resolvable -- it's painful, and
possibly impossible, e.g. w/ internal types -- so instead we should "demark"
the unresolvable types.

`.params.txt` output will use `.*` as a type prefix, e.g.

	method(.*UnresolvableType foo, int bar);

`docs.xml` will output `L.*UnresolvableType;`.

Fix JavaParameterNamesLoader.cs to support the above.

[0]: https://docs.microsoft.com/en-us/xamarin/android/internals/api-design#events-and-listeners
[1]: https://github.com/xamarin/xamarin-android/tree/d48cf04f9749664bf48fc16bcb920d5d941cccab/tools/javadoc2mdoc
[2]: http://docs.go-mono.com/?link=man%3amdoc(5)
[3]: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/xmldoc/
[4]: http://docs.go-mono.com/?link=man%3amdoc-export-msxdoc(1)
[5]: https://javaparser.org
[6]: https://github.com/xamarin/java.interop/blob/93df5a200e7b6f1b5add451aff66bbcb24293720/src/Xamarin.Android.Tools.Bytecode/JavaParameterNamesLoader.cs#L45-L68

Author: jonpryor

.editorconfig

@@ -30,7 +30,7 @@ indent_style = space
 indent_size = 4
 
 # Code files
-[*.{cs,csx,vb,vbx}]
+[*.{cs,csx,java,vb,vbx}]
 insert_final_newline = true
 indent_style = tab
 tab_width = 8

.vscode/settings.json

@@ -1,4 +1,5 @@
 {
+    "java.configuration.updateBuildConfiguration": "automatic",
     "nxunitExplorer.nunit": "packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe",
     "nxunitExplorer.modules": [
         "bin/TestDebug/generator-Tests.dll",

Directory.Build.props

@@ -43,6 +43,9 @@
     <XamarinAndroidToolsDirectory   Condition=" '$(XamarinAndroidToolsDirectory)' == '' ">$(MSBuildThisFileDirectory)external\xamarin-android-toolsXamarinAndroidToolsDirectory>
   PropertyGroup>
   <PropertyGroup>
+    <GradleHome Condition=" '$(GradleHome)' == '' ">$(MSBuildThisFileDirectory)build-tools\gradleGradleHome>
+    <GradleWPath Condition=" '$(GradleWPath)' == '' ">$(GradleHome)\gradlewGradleWPath>
+    <GradleArgs Condition=" '$(GradleArgs)' == '' ">--stacktrace --no-daemonGradleArgs>
     <JavacSourceVersion Condition=" '$(JavacSourceVersion)' == '' ">1.8JavacSourceVersion>
     <JavacTargetVersion Condition=" '$(JavacTargetVersion)' == '' ">1.8JavacTargetVersion>
     <_BootClassPath Condition=" '$(JreRtJarPath)' != '' ">-bootclasspath "$(JreRtJarPath)"_BootClassPath>

Java.Interop.code-workspace

@@ -4,5 +4,7 @@
  			"path": "."
  		}
  	],
- 	"settings": {}
+	"settings": {
+		"java.configuration.updateBuildConfiguration": "interactive"
+	}
  }

Java.Interop.sln

@@ -91,6 +91,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Java.Interop.Tools.Generato
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Interop.Tools.Generator-Tests", "tests\Java.Interop.Tools.Generator-Tests\Java.Interop.Tools.Generator-Tests.csproj", "{7F4828AB-3908-458C-B09F-33C74A1368F9}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "java-source-utils", "tools\java-source-utils\java-source-utils.csproj", "{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}"
+EndProject
 Global
 	GlobalSection(SharedMSBuildProjectFiles) = preSolution
 		src\Java.Interop.NamingCustomAttributes\Java.Interop.NamingCustomAttributes.projitems*{58b564a1-570d-4da2-b02d-25bddb1a9f4f}*SharedItemsImports = 5
@@ -251,6 +253,10 @@ Global
 		{7F4828AB-3908-458C-B09F-33C74A1368F9}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{7F4828AB-3908-458C-B09F-33C74A1368F9}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{7F4828AB-3908-458C-B09F-33C74A1368F9}.Release|Any CPU.Build.0 = Release|Any CPU
+		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -294,6 +300,7 @@ Global
 		{0E3AF6C1-7638-464D-9174-485D494499DC} = {C8F58966-94BF-407F-914A-8654F8B8AE3B}
 		{C2FD2F12-DE3B-4FB9-A0D3-FA3EF597DD04} = {0998E45F-8BCE-4791-A944-962CD54E2D80}
 		{7F4828AB-3908-458C-B09F-33C74A1368F9} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
+		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74} = {C8F58966-94BF-407F-914A-8654F8B8AE3B}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {29204E0C-382A-49A0-A814-AD7FBF9774A5}

Makefile

@@ -45,6 +45,7 @@ run-all-tests:
 	$(MAKE) run-tests                 || r=1 ; \
 	$(MAKE) run-test-jnimarshal       || r=1 ; \
 	$(MAKE) run-ptests                || r=1 ; \
+	$(MAKE) run-java-source-utils-tests     || r=1 ; \
 	exit $$r;
 
 include build-tools/scripts/msbuild.mk
@@ -144,6 +145,9 @@ run-ptests: $(PTESTS) bin/Test$(CONFIGURATION)/$(JAVA_INTEROP_LIB)
 	$(foreach t,$(PTESTS), $(call RUN_TEST,$(t))) \
 	exit $$r;
 
+run-java-source-utils-tests:
+	$(MSBUILD) $(MSBUILD_FLAGS) tools/java-source-utils/java-source-utils.csproj /t:RunTests
+
 bin/Test$(CONFIGURATION)/$(JAVA_INTEROP_LIB): bin/$(CONFIGURATION)/$(JAVA_INTEROP_LIB)
 	cp $< $@
 

build-tools/Java.Interop.BootstrapTasks/Java.Interop.BootstrapTasks/JdkInfo.cs

@@ -56,8 +56,8 @@ public override bool Execute ()
 			Directory.CreateDirectory (Path.GetDirectoryName (PropertyFile.ItemSpec));
 			Directory.CreateDirectory (Path.GetDirectoryName (MakeFragmentFile.ItemSpec));
 
-			WritePropertyFile (jdk.JarPath, jdk.JavacPath, jdk.JdkJvmPath, rtJarPath, jdk.IncludePath);
-			WriteMakeFragmentFile (jdk.JarPath, jdk.JavacPath, jdk.JdkJvmPath, rtJarPath, jdk.IncludePath);
+			WritePropertyFile (jdk.JavaPath, jdk.JarPath, jdk.JavacPath, jdk.JdkJvmPath, rtJarPath, jdk.IncludePath);
+			WriteMakeFragmentFile (jdk.JavaPath, jdk.JarPath, jdk.JavacPath, jdk.JdkJvmPath, rtJarPath, jdk.IncludePath);
 
 			return !Log.HasLoggedErrors;
 		}
@@ -93,7 +93,7 @@ Action CreateLogger ()
 			return logger;
 		}
 
-		void WritePropertyFile (string jarPath, string javacPath, string jdkJvmPath, string rtJarPath, IEnumerable<string> includes)
+		void WritePropertyFile (string javaPath, string jarPath, string javacPath, string jdkJvmPath, string rtJarPath, IEnumerable<string> includes)
 		{
 			var msbuild = XNamespace.Get ("http://schemas.microsoft.com/developer/msbuild/2003");
 			var project = new XElement (msbuild + "Project",
@@ -104,6 +104,10 @@ void WritePropertyFile (string jarPath, string javacPath, string jdkJvmPath, str
 						new XElement (msbuild + "ItemGroup",
 							includes.Select (i => new XElement (msbuild + "JdkIncludePath", new XAttribute ("Include", i)))))),
 				new XElement (msbuild + "PropertyGroup",
+					new XElement (msbuild + "JavaSdkDirectory", new XAttribute ("Condition", " '$(JavaSdkDirectory)' == '' "),
+						JavaHomePath),
+					new XElement (msbuild + "JavaPath", new XAttribute ("Condition", " '$(JavaPath)' == '' "),
+						javaPath),
 					new XElement (msbuild + "JavaCPath", new XAttribute ("Condition", " '$(JavaCPath)' == '' "),
 						javacPath),
 					new XElement (msbuild + "JarPath", new XAttribute ("Condition", " '$(JarPath)' == '' "),
@@ -121,10 +125,11 @@ static XElement CreateJreRtJarPath (XNamespace msbuild, string rtJarPath)
 					rtJarPath);
 		}
 
-		void WriteMakeFragmentFile (string jarPath, string javacPath, string jdkJvmPath, string rtJarPath, IEnumerable<string> includes)
+		void WriteMakeFragmentFile (string javaPath, string jarPath, string javacPath, string jdkJvmPath, string rtJarPath, IEnumerable<string> includes)
 		{
 			using (var o = new StreamWriter (MakeFragmentFile.ItemSpec)) {
 				o.WriteLine ($"export  JI_JAR_PATH          := {jarPath}");
+				o.WriteLine ($"export  JI_JAVA_PATH         := {javaPath}");
 				o.WriteLine ($"export  JI_JAVAC_PATH        := {javacPath}");
 				o.WriteLine ($"export  JI_JDK_INCLUDE_PATHS := {string.Join (" ", includes)}");
 				o.WriteLine ($"export  JI_JVM_PATH          := {jdkJvmPath}");

build-tools/gradle/gradle/wrapper/gradle-wrapper.jar

@@ -0,0 +1,5 @@
+distributionBase=GRADLE_USER_HOME
+distributionPath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-5.1-bin.zip
+zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists