Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use forked gems with smaller footprint #112

Merged
merged 1 commit into from
Jan 14, 2025
Merged

Conversation

Fryguy
Copy link
Member

@Fryguy Fryguy commented Oct 7, 2024

This PR uses a fork of the azure-sdk-for-ruby with unused APIs and profiles removed.

This builds on the https://github.com/ManageIQ/azure-sdk-for-ruby/tree/azure-sdk-for-ruby_production branch which adds a commit with a script for dropping gem sizes, then executes that script and commits it as the next 8 commits. I've tested this locally with the azure stack PR and all of the tests pass, and additionally I have tried to autoload every autoloadable constant and that also works.

Time to load:

azure_mgmt_network
  4.195398   1.299338   5.494736 (  5.541836)   # Before - Initial autoload everything
  0.662541   0.218067   0.880608 (  0.889964)   # After - Autoload after purge unused APIs
  0.313432   0.132122   0.445554 (  0.462570)   # After - Autoload after purge profiles

Disk usage:

Before:
 26M	azure_mgmt_compute-0.22.0
3.5M	azure_mgmt_monitor-0.19.0
140M	azure_mgmt_network-0.26.1
8.4M	azure_mgmt_resources-0.18.2
177.9M	total

After:
2.5M	azure_mgmt_compute-0.22.0.1
2.0M	azure_mgmt_monitor-0.19.0.1
4.5M	azure_mgmt_network-0.26.1.1
1.0M	azure_mgmt_resources-0.18.2.1
10.0M	total (167.9MB / 94% smaller)

The high level way the script works is that it comments out all of the "require" lines for the gem in question, then tries to eager load every profile and every autoloadable constant under that profile recursively. When it blows up, it captures which constant was missing, uncomments that line, and tries again. It keeps doing this until the gem no longer blows up. Then, everything that's still commented out is clearly not needed, so it deletes all those require lines as well as the api directories. This script is written generically, so if we need to add a new gem in the future, or we need to include a newer profile, it should be trivial to re-purge it with this script, restoring what was deleted or deleting what is newly no longer needed.

Closes ManageIQ/manageiq-pods#737

@Fryguy
Copy link
Member Author

Fryguy commented Oct 7, 2024

Weird - this all works locally, but for some reason, the git-based gem doesn't work. In the end, we're going to probably use a fork and release a gem on our gems repo but I wanted to see tests passing in CI first.

@miq-bot
Copy link
Member

miq-bot commented Jan 13, 2025

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s).

@Fryguy Fryguy closed this Jan 13, 2025
@Fryguy Fryguy reopened this Jan 13, 2025
@Fryguy Fryguy closed this Jan 13, 2025
@Fryguy Fryguy reopened this Jan 13, 2025
@Fryguy Fryguy closed this Jan 13, 2025
@Fryguy Fryguy reopened this Jan 13, 2025
@Fryguy Fryguy removed the stale label Jan 13, 2025
@Fryguy Fryguy closed this Jan 13, 2025
@Fryguy Fryguy reopened this Jan 13, 2025
@Fryguy Fryguy changed the title [WIP] Use drop_unused branch [WIP] Use forked gems with smaller footprint Jan 13, 2025
@Fryguy
Copy link
Member Author

Fryguy commented Jan 13, 2025

I've made an official fork on the gem at https://github.com/ManageIQ/azure-sdk-for-ruby/tree/azure-sdk-for-ruby_production. We're looking to publish these gems on https://rubygems.manageiq.org soon, and then I can un-WIP

@Fryguy Fryguy changed the title [WIP] Use forked gems with smaller footprint Use forked gems with smaller footprint Jan 13, 2025
@Fryguy
Copy link
Member Author

Fryguy commented Jan 13, 2025

@agrare This is ready to go.

One thing I also want to do is update the core repo with the source for these forked gems, but this can be merged independent of that. Technically it will work as is, but there's a potential security issue where if someone got a hold of the azure_mgmt_computer gem, and pushed this same version number, then it would take precendence, and be chosen instead. There's no "nice" way to do it in this repo as part of the gemspec file, because gemspecs can't do sources. So, if we put it in the core repo, then this repo also gets it by extension of pull in spec/manageiq.

@Fryguy Fryguy added performance and removed wip labels Jan 13, 2025
@Fryguy
Copy link
Member Author

Fryguy commented Jan 14, 2025

For reference, we will have a similar issue with the python library for the ansible venv. Here's the issue related to that one: Azure/azure-sdk-for-python#17801

@agrare agrare merged commit e84ce52 into ManageIQ:master Jan 14, 2025
3 of 4 checks passed
@Fryguy Fryguy deleted the drop_unused branch January 14, 2025 14:28
Fryguy added a commit to Fryguy/manageiq that referenced this pull request Jan 14, 2025
Forked gems were added to the azure stack plugin in ManageIQ/manageiq-providers-azure_stack#112
to reduce footprint by nearly 170MB.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce azure gem
3 participants