Add an option to disable asset git/GitHub API requests

[schmunk42]

Continued from yiisoft/yii2#8688 (comment)

Proposal: Add an option, like --skip-asset-info-update or ENV variable FXP_SKIP_ASSET_INFO_UPDATE to disable git and/or GitHub API requests, which query for new versions/tags, since this might take a while.

The option should only have an effect, when local information about the repository already exists. If not the plugin should download the information.

[francoispluchino]

Add command option is not possible, so only env variable is possible.

[schmunk42]

Ah sorry, I mixed that up with new commands, but I think it does not make sense (or maybe it's not even possible) to have something like composer update-skip-asset-info.

[francoispluchino]

This, it's possible.

[schmunk42]

So I looked a bit into this and if you skip updating the remote git server the performance gain are about ~30-50% (!). I.e. commenting this line.

Updates still worked if the cache is filled.

Another huge improvement would be to load bower.json files also from cache, this removes another 0,5-1 seconds for every bower package from the total time taken for an update:

[89.2MB/9.38s] Downloading https://bower.herokuapp.com/packages/select2
[89.2MB/10.05s] Writing /Users/tobias/.composer/cache/repo/https---bower.herokuapp.com-packages/select2-ca22abe1ad1684dd62ec875a0b5c25a533cf287a-package.json into cache

Looks to me like the file is always downloaded and then written to the cache.

Maybe this could also require some changes in composer itself. Do you think it's worth to create an issue like "Feature: Allow updates from cache only"?

Last but not least it would be awesome if Git updates could be run in parallel, like downloads with prestissimo. I think this would make the above changes obsolete anyway.

[schmunk42]

This could be combined with a functionality checking an expire setting, like: "Warning Asset package definitions have not been updated for 7 days".

FXP_SKIP_ASSET_INFO_UPDATE_DAYS=7

While I'd also like a separate command, the ENV variable is the way to set a default.

@francoispluchino Any advice where to start or do you have plans about this feature, it would be great improvement.

---

related: yiisoft/yii2#13064

[francoispluchino]

The bower.json and package.json files are cached. It only has files in the main branch that are not cached.

Regarding the update-skip-asset-info command, Maybe prefer a namespace fxp:update (plus court), And it would only configure the variable FXP_SKIP_ASSET_INFO_UPDATE_DAYS (via an option, 7 by default), and use the same code and options as the update command.

You can make a pull request for this feature!

[schmunk42]

I like the syntax fxp:update - using it in phd since a while :) The only thing is that you actually don't do a fxp update, rather the opposite, like (update-fast, fxp:update-quick)

[schmunk42]

So, @handcode and I dug a bit on this ...

To really get a huge performance gain, we'd need to address two things.

One is this https://github.com/fxpio/composer-asset-plugin/blob/master/Repository/BowerRepository.php#L58
Here the asset plugin tells composer where it can find the git repo for an extension.

We hacked it like so:

    // fake local URL, if exists
    $local = $data['url'];
    $local = str_replace('/','-', $local);
    $local = str_replace('https:','/path/to/user/.composer/cache/vcs/https-', $local);
    if (is_dir($local)) {
        $data['url'] = $local;
    }

This skips the update of the git repository, if it exists, actually by setting the URL to a local path.

---

The other one is the URL of the repo itself, see https://github.com/fxpio/composer-asset-plugin/blob/master/Repository/BowerRepository.php#L34

return 'file:///path/to/bower-definition.json';

This one is harder to fake, because we don't know the hash of the repo. The files we need would be in ~/.composer/cache/repo/https---bower.herokuapp.com-packages.

[update] Would also be great if we could just delegate this to prestissimo, that might also be sufficient.

---

So, both points mentioned above would essentially skip the update of asset repositories.
Some benchmarks with a really huge project: 336 packages (55 of them are from bower):

• current default behavior, including github-api-requests ~300-500 secs (honestly, I didn't even let it finish)
• without API requests ~80 secs
• with both hotfixes from above <20 secs (!!!)

Would love to hear some feedback about this.

[francoispluchino]

@schmunk42 Do not forget NPM Repository. Otherwise, your idea is interesting.

Regarding the manipulation of the plugin options, I have already transferred the configuration of the plugin (formerly extra.asset-*) in the config.fxp-asset.* section. Because the config section doesn't change the hash of the locked file composer.lock, which is not the case for the extra section.

Furthermore, I will add support for the global configuration (#273), as well as support of the environment variables to change the configuration, to override the configuration with formatting: FXP_ASSET__<VARIABLE_NAME_WITH_UPPERCASES_AND_UNDERSCORES> (#274).

In this way you will can propose your solution to disable the analysis of the asset dependencies.

[schmunk42]

Do not forget NPM Repository.

I haven't tested this, but it looked to me, like this would also be the way npm works with fxp, simply because composer works that way.

[francoispluchino]

Your last comment change the behavior directly on the BowerRepository class, so, it's independant of the Composer, and the change is only for Bower in this plugin.

[schmunk42]

So, I played around a bit with npm-assets ...

One thing I noticed is, that despite the fact npm is handing out a lot more information than bower, it takes only about 0,2 secs to get that information compared to 0,6 secs for bower (which is only the repo info) - I am speaking of Downloading https://bower.herokuapp.com/packages/bootstrap vs. Downloading https://registry.npmjs.org/bootstrap - these are pretty rough numbers, but they sum up if you have a lot of asset packages.

The rest of the processes are the same, pruning and updating the repo takes the most amount of time, usually ~1-3 secs per asset repository (depending on the size)

Executing command (/Users/tobias/.composer/cache/vcs/https---github.com-twbs-bootstrap.git/): git remote set-url origin 'https://github.com/twbs/bootstrap.git' && git remote update --prune origin

This could also be almost completely eliminated when skipping that check by using a local repo. Another option would be to optimize that step, but that would also involve composer itself.

CC: @tonydspaniard @cebe @samdark @mikehaertl

[schmunk42]

Some more findings ...

The line in composer which runs the update is here.
If this line would be missing origin (or if it would be configurable) - we could do something like this:

Taken from https://git-scm.com/docs/git-remote - just configure that origin has not to be updated - or do that manually.

I know that this somehow defeats the purpose of "update" but on the other hand, you do not have to update & purge your local mirror every time - other package manager warn you, if this hasn't been done for some time (eg. two weeks on Macports).

I hardy dare to CC: @Seldaek - but I'd really like to hear his opinion about it.