Add path to global.json SDK version lock

[rainersigwald]

Builders of large systems often want to be able to explicitly specify an SDK version and avoid installing the CLI to a machine-global location.

Currently, global.json allows the version lock, but the specified version must be installed in either a machine-global location or a nonstandard location must be specified by the environment variable DOTNET_MSBUILD_SDK_RESOLVER_SDKS_DIR.

That environment means that you must launch Visual Studio from a specific environment to get the downloaded/private SDKs.

There could be an extension to the SDK resolver to respect a path specified in the global.json. Something like

{
    "sdk": {
        "version": "1.0.0",
        "path": "tools/downloadedsdk"
  }
}

could be equivalent to setting DOTNET_MSBUILD_SDK_RESOLVER_SDKS_DIR=%GlobalJsonPath%\tools\downloadedsdk before invoking the resolver.

This would be visible by any invocation (dotnet, msbuild, VS, or MSBuild API) since it's file-based.

[rainersigwald]

@nguerrera I think this might be a cleaner way to address @jaredpar's scenario from dotnet/msbuild#2095.

[jaredpar]

I'm not sure this would work for Roslyn: we don't use the CLI to build. The CLI won't be able to build Roslyn anytime in the foreseeable future due to the number of desktop specific MSBuild extensions our build contains: WPF, VS SDK, SWIX, etc ...

[nguerrera]

@rainersigwald This is exactly what I had in mind when I wrote:

(Now, that actually overlaps with another feature that's evolving and we will likely land in a place where you can edit global.json to get this behavior without setting any environment variables, but the mechanism here applies more generally to arbitrary resolvers with arbitrary input.)

There wasn't an issue for this yet (thanks for starting one), just mail threads so far, with a meeting scheduled soon to hash out the details. There's an existing mechanism called "multilevel lookup" in hostfxr that adds a well-known user profile location to the sdk search. This would be an evolution of that.

I did not expect dotnet/msbuild#2095 to be controversial, but I understand your concerns now. I wasn't intending for this and that to be mutually exclusive. I was imagining that being able to pass arbitrary data down to arbitrary resolvers would be generally useful.

@jaredpar

I'm not sure this would work for Roslyn: we don't use the CLI to build.

The resolver called by desktop msbuild will respect global.json too. That's part of it's main purpose: to pick the same msbuild targets for VS that would be used by the CLI msbuild invoked on the same project.

Do you build with CLI on Mac/Linux? If your process installs .NET Core SDK (official name of entire CLI not just dotnet/sdk) to a local location and sets this feature up via global.json, then you can have one mechanism for pinning down the precise set of things that come from the .NET Core SDK across platforms, and not require it to be globally installed anywhere. On Windows, you'd only use the tasks and targets, but if somebody wanted to use dotnet on the portable projects on Windows it would work. That could give you a simple way to reproduce issues that only happen on Mac or Linux official builds on a Windows dev box.

[nguerrera]

This could be equivalent to setting DOTNET_MSBUILD_SDK_RESOLVER_SDKS_DIR

global.json selects the full .NET Core SDK (CLI) used, not just the msbuild SDKs. We don't even have a resolver when we're invoked via dotnet as it will have taken us to a universe where we have matching msbuild SDKs for the given version.

So global.json should instead specify equivalent of DOTNET_MSBUILD_SDK_RESOLVER_CLI_DIR (matching layout of C:\Program Files\dotnet). More granular control over the resolver than that has to go somewhere other than global.json.

[jaredpar]

Can I get more information on global.json: in particular where it lives, how it's found, etc ...?

Do you build with CLI on Mac/Linux? I

Yes and depending on the answer above it seems like a problem. The path to the downloaded SDK is going to be very different between the two systems.

{
    "sdk": {
        "version": "1.0.0",
        "path": "tools/downloadedsdk"
  }
}

In order for this to solve the problem the SDK essentially has to be within the cone of the global.json. Guessing that mean that it has to be within the cone of the repo. No other tool has that requirement for our build and it's going to take some doing to make that work.

Still overall feels clunky. MSBuild properties can today define everything about my build. Makes it dynamic, very easy to manage. Now we have JSON (again) and I have to write generators to translate between my MSBuild files and this new JSON file.

[tmat]

In order for this to solve the problem the SDK essentially has to be within the cone of the global.json. Guessing that mean that it has to be within the cone of the repo. No other tool has that requirement for our build and it's going to take some doing to make that work.

I think this will be easy to make work in our repos. The only thing that needs to be restored to a directory specified in global.json is the immediate SDK used by the projects (which is not necessarily .NET SDK). In RepoToolset repos this is RepoToolset SDK (projects use <Project Sdk="RoslynTools.RepoToolset"/>).

Our current global.json for symreader repo looks like so:

{
  "sdk": {
    "version": "2.1.100-preview-007366"
  },
  "msbuild-sdks": {
    "RoslynTools.RepoToolset": "1.0.0-beta2-62705-02"
  }
}

Let's say we could add the path like so:

{
  "sdk": {
    "version": "2.1.100-preview-007366"
  },
  "msbuild-sdks": {
    "RoslynTools.RepoToolset": { "version": "1.0.0-beta2-62705-02", "path": ".sdk" }
  }
}

and this would restore the RepoToolset to {repo-root}\.sdk directory (next to global.json)

The RepoToolset restores .NET SDK by importing it in its sdk\Sdk.props file:

<Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />

at which point we can use arbitrary msbuild properties to specify the path to restore to.

[jaredpar]

@tmat that makes sense. I think that's workable.

[abelbraaksma]

This would be a very welcome addition to the rather inflexible global.json config options. Currently, not being able to point to a local path, and the VS installer constantly uninstalling previous SDK versions, makes it pretty hard to do any decent SDK-dependent development (like the F# repo project files, see dotnet/fsharp#10193, which should be fixed on 3.1.302, but the VS installer removes this in favor of 3.1.402).

Since it is trivial to create an msbuild script that downloads a specific SDK version and installs it locally, it would be very, very nice if such behavior could be baked into global.json as well:

• specify a local path, or just use token "local": true and standardize the path to be .dotnet
• if SDK is absent in that location, VS installs the SDK automatically upon fist opening the solution

All in all, that would create a much better experience then the now rather hard-to-understand "cannot load project or solution" message, which baffles newcomers and experienced users alike...

[RussKie]

This will have a significant positive impact on the developer experience for .NET repositories, such as https://github.com/dotnet/runtime/, https://github.com/dotnet/winforms, etc. Currently it is impossible to double click on a solution file to open the solutions in VS, because we build against nightly SDKs, which are located in custom locations and are forever changing.

[KirillOsenkov]

As a workaround, I keep around a .bat file with this example content:

set PATH=C:\msbuild\.dotnet;%PATH%
set DOTNET_INSTALL_DIR=C:\msbuild\.dotnet
set DOTNET_MULTILEVEL_LOOKUP=0

If you go to the repo directory (C:\msbuild in my case) and run this .bat file, then run devenv.exe mysolution.sln, it will open with these environment variables set and the SDK resolution by MSBuild will use the .dotnet that's private to the repo.

[RussKie]

That's what other dotnet repos are doing AFAIK in some shape or form. And this is not the best developer experience. It also makes it hard to run ad-hoc tests...

[KirillOsenkov]

Oh, you don't have to convince me ;) I know it's terrible and I've spilled blood, sweat and tears to even arrive at this workaround by debugging deep into the SDK resolution process.

Lots of pain and friction caused by this over the years.

[RussKie]

To include other points form offline chats and summarise:

There a number of use cases that really benefit from this (or comparable) functionality:

1. Developer experience building and contributing to .NET repos, such as https://github.com/dotnet/runtime/, https://github.com/dotnet/winforms, etc. Currently it is impossible to double click on a solution file to open the solutions in VS, because we build against nightly SDKs, which are located in custom locations and are forever changing. This make it unnecessary hard and complicated for our (especially new) contributors. Right now a lot of our tooling is cli-centric (which may be expected for non Windows platforms and CI/CD scenarios), but our Windows/Visual Studio devex is very cumbersome. Repos have custom scripts or instructions that must be run in order to open solutions in Visual Studio.
2. Local testing - it is very difficult to run customer repos against the nightlies. These builds must either be installed globally, or one has to jump through hoops and run custom scripts to bootstrap a sample.

If it is difficult to provide the download functionality, we could probably do this in staggered approach, i.e. provide the ability to resolve an SDK from a custom location.

• If the path is not specified - continue doing what we do today.
• If the path is invalid - fail.

Maybe add another property to specify an error message that can describe how to bootstrap or where to download from. E.g.:

{
  "sdk": {
    "version": "2.1.100-preview-007366"
  },
  "msbuild-sdks": {
    "RoslynTools.RepoToolset": { 
      "version": "1.0.0-beta2-62705-02",
      "path": ".sdk",
      "instructions": "Run .\restore.cmd to download the latest SDK.",
    }
  }
}