-
-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert usermods to static libraries #4480
Draft
willmmiles
wants to merge
66
commits into
wled:main
Choose a base branch
from
willmmiles:usermod-libs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
b8685f2
Convert usermods to static libraries
willmmiles 4d5e0ca
load_usermods: Expand name search
willmmiles 71b0e8e
Convert AnimARTrix usermod to library
willmmiles 90b1815
Convert usermods from header to library
netmindz a5575bc
Fix naming of usermod files to match library name
netmindz 6e76a72
Convert usermods from header to library
netmindz 67022be
Build all usermods
netmindz cbed841
Include basic usermods in CI
netmindz c16d83f
Build custom_usermods = audioreactive auto_save animartrix
netmindz d64cedd
Build custom_usermods = audioreactive auto_save animartrix
netmindz 2381e32
Define dependencies for ADS1115 usermod
netmindz 79bac91
use bigger partitions for usermods env
netmindz d3eec72
Defining more usermod dependencies
netmindz 3521732
fix env:userods
netmindz 52b784e
fix env:usermods
netmindz 52bee88
fix word_clock_matrix naming
netmindz 075fd4d
Defining more usermod dependencies
netmindz 4c19341
Merge branch 'usermod-libs-migration' of github.com:netmindz/WLED int…
netmindz 8527d23
audioreactive: Move flags to library
willmmiles cc9e9b1
Fix usermod library builds
willmmiles f2626b0
Remove now-obsolete AR_build_flags
willmmiles 24accf9
Remove now redundant build section from library.json
netmindz b380d5e
reinstate libArchive:false
netmindz 2c9c413
Merge branch 'usermod-libs' into usermod-libs-migration
netmindz 1dbd706
Revert LOROL_LITTLEFS hack
netmindz e471487
typo in usermods/sensors_to_mqtt/library.json
netmindz 869e275
typo in usermods/sensors_to_mqtt/library.json
netmindz 0b8721c
Fix usermod libArchive setting
willmmiles 270d75a
Update usermod deps earlier
willmmiles 8fd9052
Integrate usermods environment
willmmiles 30559cd
Fix dependency for EleksTube_IPS usermod
willmmiles 32607ee
Revert incorrect testing platformio.ini
willmmiles ef2eb07
Merge branch 'main' into usermod-libs
2adf745
Update env:usermods to use V4
5da380e
Update dependencies for sensors_to_mqtt
59a79a3
Add deps for usermods/BME280_v2
0afd2fe
Destructor must be public
5d05d79
Add usermod dependencies
8487dd7
Disable build of usermods that are broken at the moment
15edfcd
Fix usermod platformio integration
willmmiles 193926c
usermods: Remove #pragma once from cpps
willmmiles 2b07be1
usermod mpu6050: Fix incorrect int type
willmmiles 44a1a1e
usermods: Fix MQTT checks
willmmiles 48372bc
platformio: Fix audioreactive usermod for ESP8266
willmmiles 0233dae
usermods/sensors_to_mqtt: Fix char type
willmmiles 30a697e
usermods/PWM_fan: Disable pending cross-mod check
willmmiles 1d558e8
platformio.ini: Add extra flags for usermod test
willmmiles 0d44e7e
Usermods: Remove libArchive
willmmiles 86f9761
platformio.ini: Fix esp32dev_V4 usermods
willmmiles 5d392d8
load_usermods: Improve include path assembly
willmmiles 4bc3408
load_usermods: Don't cross usermod includes
willmmiles a5b972d
Merge remote-tracking branch 'upstream/main' into usermod-libs
willmmiles 51db63d
load_usermods: Also search for mod_v2
willmmiles 851e9ec
Remove deprecated mqtt_switch_v2 usermod
willmmiles 070b08a
Rename usermod EXAMPLE_v2 to EXAMPLE
willmmiles b3f9983
First half of usermod readme updates
willmmiles 7a40ef7
Fix up PWM_fan
willmmiles 2eff6b7
usermod/sensors_to_mqtt: Add explicit dep
willmmiles 1688546
Fix RTC usermod
willmmiles f72b5d0
usermod/pixels_dice_try: Add missing dep
willmmiles 2431f20
load_usermods: Split on any whitespace
willmmiles d0b5997
Fix up BME280_v2 usermod
willmmiles e6910f7
Disable EleksTube_IPS usermod
willmmiles c57be77
Fix sensor usermod globals
willmmiles 078a054
usermods/pixels_dice_tray: Fix BLE dependency
willmmiles 1dc7647
Merge branch 'main' into usermod-libs
netmindz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| Import('env') | ||
| import os.path | ||
| from collections import deque | ||
| from pathlib import Path # For OS-agnostic path manipulation | ||
| from platformio.package.manager.library import LibraryPackageManager | ||
|
|
||
| usermod_dir = Path(env["PROJECT_DIR"]) / "usermods" | ||
| all_usermods = [f for f in usermod_dir.iterdir() if f.is_dir() and f.joinpath('library.json').exists()] | ||
|
|
||
| if env['PIOENV'] == "usermods": | ||
| # Add all usermods | ||
| env.GetProjectConfig().set(f"env:usermods", 'custom_usermods', " ".join([f.name for f in all_usermods])) | ||
|
|
||
| def find_usermod(mod: str): | ||
| """Locate this library in the usermods folder. | ||
| We do this to avoid needing to rename a bunch of folders; | ||
| this could be removed later | ||
| """ | ||
| # Check name match | ||
| mp = usermod_dir / mod | ||
| if mp.exists(): | ||
| return mp | ||
| mp = usermod_dir / f"{mod}_v2" | ||
| if mp.exists(): | ||
| return mp | ||
| mp = usermod_dir / f"usermod_v2_{mod}" | ||
| if mp.exists(): | ||
| return mp | ||
| raise RuntimeError(f"Couldn't locate module {mod} in usermods directory!") | ||
|
|
||
| usermods = env.GetProjectOption("custom_usermods","") | ||
| if usermods: | ||
| # Inject usermods in to project lib_deps | ||
| proj = env.GetProjectConfig() | ||
| deps = env.GetProjectOption('lib_deps') | ||
| src_dir = proj.get("platformio", "src_dir") | ||
| src_dir = src_dir.replace('\\','/') | ||
| mod_paths = {mod: find_usermod(mod) for mod in usermods.split()} | ||
| usermods = [f"{mod} = symlink://{path}" for mod, path in mod_paths.items()] | ||
| proj.set("env:" + env['PIOENV'], 'lib_deps', deps + usermods) | ||
| # Force usermods to be installed in to the environment build state before the LDF runs | ||
| # Otherwise we won't be able to see them until it's too late to change their paths for LDF | ||
| # Logic is largely borrowed from PlaformIO internals | ||
| not_found_specs = [] | ||
| for spec in usermods: | ||
| found = False | ||
| for storage_dir in env.GetLibSourceDirs(): | ||
| #print(f"Checking {storage_dir} for {spec}") | ||
| lm = LibraryPackageManager(storage_dir) | ||
| if lm.get_package(spec): | ||
| #print("Found!") | ||
| found = True | ||
| break | ||
| if not found: | ||
| #print("Missing!") | ||
| not_found_specs.append(spec) | ||
| if not_found_specs: | ||
| lm = LibraryPackageManager( | ||
| env.subst(os.path.join("$PROJECT_LIBDEPS_DIR", "$PIOENV")) | ||
| ) | ||
| for spec in not_found_specs: | ||
| #print(f"LU: forcing install of {spec}") | ||
| lm.install(spec) | ||
|
|
||
|
|
||
| # Utility function for assembling usermod include paths | ||
| def cached_add_includes(dep, dep_cache: set, includes: deque): | ||
| """ Add dep's include paths to includes if it's not in the cache """ | ||
| if dep not in dep_cache: | ||
| dep_cache.add(dep) | ||
| for include in dep.get_include_dirs(): | ||
| if include not in includes: | ||
| includes.appendleft(include) | ||
| if usermod_dir not in Path(dep.src_dir).parents: | ||
| # Recurse, but only for NON-usermods | ||
| for subdep in dep.depbuilders: | ||
| cached_add_includes(subdep, dep_cache, includes) | ||
|
|
||
| # Monkey-patch ConfigureProjectLibBuilder to mark up the dependencies | ||
| # Save the old value | ||
| old_ConfigureProjectLibBuilder = env.ConfigureProjectLibBuilder | ||
|
|
||
| # Our new wrapper | ||
| def wrapped_ConfigureProjectLibBuilder(xenv): | ||
| # Update usermod properties | ||
| # Set libArchive before build actions are added | ||
| for um in (um for um in xenv.GetLibBuilders() if usermod_dir in Path(um.src_dir).parents): | ||
| build = um._manifest.get("build", {}) | ||
| build["libArchive"] = False | ||
| um._manifest["build"] = build | ||
|
|
||
| # Call the wrapped function | ||
| result = old_ConfigureProjectLibBuilder.clone(xenv)() | ||
|
|
||
| # Fix up include paths | ||
| # In PlatformIO >=6.1.17, this could be done prior to ConfigureProjectLibBuilder | ||
| wled_dir = xenv["PROJECT_SRC_DIR"] | ||
| # Build a list of dependency include dirs | ||
| # TODO: Find out if this is the order that PlatformIO/SCons puts them in?? | ||
| processed_deps = set() | ||
| extra_include_dirs = deque() # Deque used for fast prepend | ||
| for dep in result.depbuilders: | ||
| cached_add_includes(dep, processed_deps, extra_include_dirs) | ||
|
|
||
| for um in [dep for dep in result.depbuilders if usermod_dir in Path(dep.src_dir).parents]: | ||
| # Add the wled folder to the include path | ||
| um.env.PrependUnique(CPPPATH=wled_dir) | ||
| # Add WLED's own dependencies | ||
| for dir in extra_include_dirs: | ||
| um.env.PrependUnique(CPPPATH=dir) | ||
|
|
||
| return result | ||
|
|
||
| # Apply the wrapper | ||
| env.AddMethod(wrapped_ConfigureProjectLibBuilder, "ConfigureProjectLibBuilder") | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The downside to this change is that you have no control of what you are building
For testing I was setting my local platform_override.ini to only list the mods I was trying to build.
I guess I can still do that, but just need to name my env somehting like
usermods_localThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not thrilled at magic environment names, but I think it's important to be able to locally replicate what the CI system does without having to run shell commands to update local state after every
git pull. I'm open to alternative suggestions!There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly less need for an "all in one" build if we just let the CI handle a matrix build for usermods, then just define the one we are working on locally in our own platform_override ?
willmmiles#3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I stand by my opinion from above -- I think it's important that what the CI system does should be easily and directly reproducible by developers. I don't think it's appropriate to have to hand hack custom files in your personal workspace to replicate what the CI does; that's just asking to get it wrong.
From a practical perspective, I'd also like it to be simple to validate my local changes before pushing to the CI. If we depend on matrix builds, that becomes almost impossible. Id be stuck waiting on the CI system any time I want to have some confidence I haven't inadvertently broken the build of some random usermod.
Finally, I don't know much about GitHub CI; I'm concerned that adding dozens of extra builds per run would be costly.