This allows updating/reinstalling of modules with the nonet flag set (retropiemenu and runcommand) when no internet connection is available.
Incorrect variable was used in comparison ($id instead of $md_id), so the has_net wasn't forced to 1.
rp_module_flags can now contain a variable comparison to enable or disable a scriptmodule.
The comparison should be in the format :\$var:cmp:val or !:\$var:cmp:val
eg. :\$__gcc_version:-lt:7 would be evaluated as [[ $__gcc_version -lt 7 ]]
This would enable a module if the comparison was true or disable if !:\$__gcc_version:-lt:7 was used.
Only global variables set before modules are loaded (eg via system.sh) are supported.
$ is escaped so variables are not evaluated when the module is sourced.
It works without, but provides less useful information in the setup menus as the variable name will not be visible.
Only configure distcc variables if a module does not have the "nodistcc" flag.
Skip builder/setup modules to avoid having distcc changes propagate to modules built from them.
This allows easily disabling distcc for certain modules that are not compatible.
Added a fallback for getting the `modified_date` of a module if the module's file is not tracked by `git`.
If the module is just copied in the `scriptmodules` folder or under `ext`, but not part of a `git` repo, use the modification date of the file.
This avoids re-building the module when a mass update is run.
Noticed this in [this forum topic](https://retropie.org.uk/forum/topic/33922/need-help-with-openbor-retropie-extra-exarkuniv/15), seems like [RetroPie-Extra](https://github.com/Exarkuniv/RetroPie-Extra) copies selected module files under `ext`, while the main repository folder is kept separately.
As with e041a0c5 we need to use the same user for running git as the owner of the repository files.
This is also needed for an update check in rp_hasNewerModule as we check the last commit date of scriptmodules in RetroPie-Setup.
This fixes setting package date for redream and not updating unless needed.
The download function already had the --location parameter, but the packaging
code for checking for a remote file / checking file date didn't.
This is used for our package information, although it's not used to check for newer versions
as we only check the remote hash of a git repository for updates.
We also used this for checking if we need to update a binary based on the last date of a change to the
module file. However as we were using author date, if a binary was built before the change was accepted, but
more recently than the change was authored, it wouldn't trigger a binary rebuild.
It makes more sense to use the date the commit was made rather than when it was authored in both cases, as
we want to know when something was last changed in the repository rather than the date of the change.
Custom binary URLs provided by __binary_url_* scriptmodule functions were ignored for platforms
we don't host general binary builds for. This meant that lr-duckstation wouldn't show binary
install options on aarch64 / x86_64.
Moved the rp_hasBinaries check logic to rp_getBinaryUrl, so we only set a URL by default when
we have binaries for that system. This is overridden as needed by install_bin_* / __binary_url_*
scriptmodule functions. The rp_hasBinary function will check for a binary if the URL is set, so
we allow overriding of our own binaries, as well as having the ability to install 3rd party binaries
for systems we don't host binaries for.
rp_hasNewerModule will force a rebuild if the module was updated since it was installed. This was added to
handle cases where we update patches or build parameters in module. Using the last commit date is more reliable
as module dates can be changed when rebasing etc. This resolves some issues with the builder module rebuilding
binaries when not needed, due to branch switches / rebases.
Create a new function isConnected in helpers.sh which calls getIPAddress, but also sets a new global
__NET_ERRMSG if not connected. This is used in setup.sh and packages.sh so we don't duplicate the same
message between files.
Split out a new function runCurl from the download function, to do lower level curl calls, but capture any
errors into __NET_ERRMSG for display in setup.sh or via the packaging functions. Use a connect timeout of 10
in the download function rather than 60 which is quite long, but keep the 60 second limit on no data transfer.
Adjust rp_remoteFileExists to use runCurl, so we can capture any error when checking a remote file,
and use this value for display in setup and in logs etc. Switch to using long versions of curl options for
readability, and add --show-error so we still output errors when being silent.
Also switch rp_getFileDate to use new runCurl function, so that any curl argument overrides are handled, and use
long format parameters also. No error handling is done here though.
Use "Last Changed Rev:" rather than "Revision:" as we want the last revision for the URL/branch/tag
rather than the last revision for the repository.
Don't just compare commits if the commit is HEAD (used for svn repos) as it's not a static commit but
references the latest commit.
Copy retropie.pkg to archive folder named as $md_id.pkg and use this in rp_hasNewerModule if no installed
package is found. Adjust the builder code to check for a newer source package to utilise this so we only
rebuild binaries as are needed.
Some users reported issues and it was taking between 5 and 6 seconds - which points to an connectivity issue
really, as it's a small HEAD request. However increasing the timeout will help for these users.
This is a quick fix, as I will look into improving the error msgs also.
When the remote repository uses an annotated Git tag, the real commit is found
by looking for the "tag^{}" reference, since the the tag ref will point to the
tag object itself, instead of the tagged commit. See gitrevisions(7).
Also moved the retropie email used for checking to __gpg_retropie_key and use this rather than __gpg_signing_key
as we hardcode the install of the actual key - so the logic didn't make sense in the case of wanting to sign
with another key. But __gpg_signing_key can now be set to an empty string to bypass signing of binaries as used
by the build scripts. This is only useful for developers/testing binary creation on systems without our signing key.
The previous changes to check for an internet connection to avoid showing options to update failed
with these modules as they have install_bin functions which are handled by rp_hasBinary but don't
require an internet connection.
To workaround this, I have added a "nonet" flag, which if set, will force has_net to 1 in setup and packages
so the install options will show, and they can be removed and reinstalled even if no network connection is available.
runcommand can still fail due to not being able to install mesa-drm - which will be reported, but it's safer to allow
the main runcommand to be installed even without mesa-drm in case of accidental removal.
Reworked rp_loadPackageInfo to clear package information in __mod_info and defaulting pkg_origin to "unknown".
The pkg_origin was set on module load before, but this isn't enough as it was remembered when removing a package,
so wouldn't be set back to the default if no package information was found. This caused installing from a section
to install from the last pkg_origin rather than the default of "unknown" if installing a package that had been removed.
The default is best handled here also, so we have less related code in other functions.
Unless there is an error, cache the return result for rp_hasBinary and rp_hasNewerModule
and clear when running rp_setModuleInfo or removing a module.
Reworked the rp_hasNewerModule code to have a single return point to allow us to cache the result.
This speeds up navigating Setup menus.
If a module chooses a specific commit via rp_module_repo (directly or via a hook) using the first 8
characters of the git hash, only compare against the first 8 characters of our installed package hash.
This should fix any cases where an older commit is used for a module, but would not match, so a git ls-remote
would be called, which would get the last commit, and an update would be forced when not needed.
If the first parameter of rp_module_repo starts with a : it the following function will be called
when setting package information during source install, and when checking for a new source version.
The function takes the parameters "get" and "check".
"get" - should return code to set any packaging variables needed for the module hook later.
The pkg_repo_extra field has been added specifically for modules to store something different from the
standard fields also
"newer" will be called with pkg_repo_* vars set, and can be used by the function hook to determine if
a new version is available. If so, the function should return 0, or 1 if no update is required.
I also split off checking remote repo hash/commit to separate function and fix up an issue with checking
remote git hash, as previously a branch or tag could have multiple matches - we now specifically grep
the results for ref/heads and ref/tags
rp_module_repo is currently in the format
TYPE URL BRANCH COMMIT
where type is file, git or svn
If rp_module_repo URL BRANCH or COMMIT starts with a : the rest is considered to be a function that
will return the URL, BRANCH, or COMMIT via an echo. This allows for using of the new rp_module_repo
by modules that use some logic to determine what branch / commit is used.
eg. Some modules will use an older branch based on the system they are running on.
These variables are resolved before use, as some modules query the net for information, so we only
want to check these just before they are used.
With binaries, if we update a scriptmodule, we can rebuild or update the date on the binaries to force
an update. However, with the source installs, if the remote repository is not updated, an update won't
happen until it does. This could be an issue in a situation where we notice a bug in the module. In this
case we check if the module date is newer than the install date - and if so, we force an update.
Previously we eval's the return of getPackageInfo so the package variables could be created
locally to the function. However, due to the use of a subshell we couldn't cache anything in a variable
for use later.
The functions is now renamed to rp_loadPackageInfo and loads the package information into __mod_info[ID/FIELD]
eg. __mod_info[ID/pkg_origin] will contain the retropie.pkg origin field for the installed module
It also uses just grep to load the data rather than our existing ini functions for performance. As it's not
considered a user editable file, performance is key here, as we need this data when traversing the
RetroPie-Setup menus.
If a full load of a retropie.pkg is done, __mod_info[ID/pkg_info] is set to 1, as a cache flag so the data can be
reused without reloading on an additional call. rp_SetPackageInfo will clear this flag
Optional parameters for rp_loadPackageInfo can choose individual fields but if used the data won't be
cached, but it can be a lot faster - we use this to grab pkg_origin for the section menus in RetroPie-Setup.
These changes allow storing of source repo information in retropie.pkg as well as the ability
to check for a new version. To do this, repository information needs to be moved to the variable
md_module_repo in the format
md_module_repo="TYPE URL BRANCH COMMIT"
Currently types of file, git, or svn are suppported.
If using file the BRANCH and COMMIT are not used.
If using svn the BRANCH should be - and the COMMIT is the SVN revision.
This information is split up and provided to the module in the form of
md_repo_type
md_repo_url
md_repo_branch
md_repo_commit
Previously if set and if md_build contained a git repository, it would reset it, to undo any patches we
may have so they get reapplied after next git update. However, if there wasn't a .git folder in md_build it would
still remove the folder, rather than leave it behind. Adjusted the logic to fix this, but this code is very old
and was a quick "hack" so needs revisiting with upcoming packaging changes.
Doing an additional grep on every module to check if it duplicated an additional module_id, added
more overhead than was ideal. However we can just do the check on 3rd party repos, which brings performance
back to how it was pretty much, and will have a slight overhead if a user has a 3rd party repo added
although it works as it is currently due to the flexibility of the unix "date" command, we should
standardise dates so we don't end up with different formats in the configs depending on webserver output etc.
Curl headers outputs as CRLF rather than LF. Remove styled output from curl (as a precaution), and
strip CR's via tr before piping to grep, but also just use grep for processing via perl regexp (-P).
Do a grep to extract the rp_module_id before sourcing a file and check if it has already been used.
We don't want additional modules overriding existing, or accidental duplication.
Store the path of a module in __mod_info(ID/path)
Display any info/error messages when launching setup as otherwise they wouldn't be seen - they are
shown otherwise at the end of retropie_packages.sh.
Used to separate 3rd party packages from RetroPie ones. Added some extra menu items in setup showing
above each type like a heading with
---- VENDOR - TYPE ----
Where:
* VENDOR is RetroPie or the name of the folder the 3rd party repo is installed in
* TYPE is the module/package type such as emulator, libretrocore, port etc
Some users have found the ordering of the menus confusing and this should help with that also.
3rd party packages always appear at the end of a section
3rd party repositories can be checked out to RetroPie-Setup/ext
Repositories should be structured so you have:
ext/REPOSITORY/scriptmodules
Modules are only read from the following subfolders of scriptmodules
* emulators
* libretrocores
* ports
* supplementary
* admin
3rd party repositories can be checked out to RetroPie-Setup/ext
Repositories should be structured so you have:
ext/REPOSITORY/scriptmodules
Modules are only read from the following subfolders of scriptmodules
* emulators
* libretrocores
* ports
* supplementary
* admin
Previously, allthough all modules were read in, we only displayed modules that were enabled for
the current system. This has caused some confusion in the past with people installing RetroPie on another
system and wondering where a package is.
Now all packages are shown via retropie_packages.sh and in setup menus with information if they are
not enabled for the current system.
