Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a PackageCompiler plugin #290

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/PkgTemplates.jl
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ export
License,
Logo,
NoDeploy,
PackageCompiler,
ProjectFile,
Readme,
RegisterAction,
Expand Down
1 change: 1 addition & 0 deletions src/plugin.jl
Original file line number Diff line number Diff line change
Expand Up @@ -361,3 +361,4 @@ include(joinpath("plugins", "citation.jl"))
include(joinpath("plugins", "documenter.jl"))
include(joinpath("plugins", "badges.jl"))
include(joinpath("plugins", "register.jl"))
include(joinpath("plugins", "package_compiler.jl"))
96 changes: 96 additions & 0 deletions src/plugins/package_compiler.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
const PACKAGECOMPILER_DEP = PackageSpec(;
name="PackageCompiler",
uuid="9b87118b-4619-50d2-8e1e-99f35a4d4d9d",
)

"""
PackageCompiler(;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to include this docstring in the docs, e.g. on docs/src/user.md in the Misc section

make_jl = "$(contractuser(default_file("build", "make.jl")))",
precompile_jl = "$(contractuser(default_file("build", "precompile.jl")))",
sysimage_name = "sysimage",
packages = :deps
)

Sets up sysimage generation via [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does PackageCompiler.jl require a certain minimum Julia version?
If so, should we document that here? e.g. in a !!! note block?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When in the top-level directory of the generated package, a system image should be created by running:
```
shell\$ julia build/make.jl
```

## Keyword Arguments
- `make_jl::AbstractString`: Template file for `make.jl`.
- `precompile_jl::AbstractString`: Template file for `precompile.jl`.
- `sysimage_name::AbstractString`: Base path to the generated sysimage; an
appropriate extension will be added depending on the OS.
- `packages`: Determines the list of packages to bake into the
sysimage (see below).

## List of packages to include in the sysimage

The `packages` keyword argument allows specifying which packages should be
included in the sysimage. Supported values are:

- `:deps`: include in the sysimage all direct dependencies of the package.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we can robustly support this option... is there anything in the PackageCompiler docs on using a pattern like this?

- `:pkg`: include in the sysimage the package itself.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `:pkg`: include in the sysimage the package itself.
- `:pkg`: include in the sysimage only the package itself.

- vector of package names, as strings or symbols: include all listed packages into the sysimage.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- vector of package names, as strings or symbols: include all listed packages into the sysimage.
- `Vector{Symbol}` of package names: include all listed packages into the sysimage.


## Examples
```
# Explicitly list packages to include into the sysimage
PackageCompiler(packages = [:Plots, :DataFrames])
PackageCompiler(packages = ["Plots", "DataFrames"])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PackageCompiler(packages = ["Plots", "DataFrames"])


# Build a sysimage containing all direct dependencies of the current package
# (this is the default)
PackageCompiler(packages = :deps)

# Build a sysimage containing the current package itself
PackageCompiler(packages = :pkg)

# Generated sysimage will be located at \$(PWD)/foo/bar/image.\$(EXT)
PackageCompiler(sysimage_name = joinpath("foo", "bar", "image"))
```
"""
@plugin struct PackageCompiler <: Plugin
make_jl::String = default_file("build", "make.jl")
precompile_jl::String = default_file("build", "precompile.jl")
sysimage_name::String = "sysimage"
packages::Union{Symbol, AbstractVector} = :deps
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

creat_sysimage seems to require packages names be Symbol, so let's just support that (https://julialang.github.io/PackageCompiler.jl/dev/refs/#PackageCompiler.create_sysimage)

Suggested change
packages::Union{Symbol, AbstractVector} = :deps
packages::Union{Symbol, Vector{Symbol}} = :deps

end

priority(::PackageCompiler, ::Function) = DEFAULT_PRIORITY - 1 # We need SrcDir to go first.

function view(p::PackageCompiler, t::Template, pkg::AbstractString)
d = Dict{String, Any}(
"PKG" => pkg,
"SYSIMAGE_NAME" => p.sysimage_name,
)

if p.packages == :deps
d["SYSIMAGE_DEPS"] = true
elseif p.packages == :pkg
d["SYSIMAGE_LIST"] = pkg
else
d["SYSIMAGE_LIST"] = p.packages
end

d
end

function hook(p::PackageCompiler, t::Template, pkg_dir::AbstractString)
pkg = basename(pkg_dir)
build_dir = joinpath(pkg_dir, "build")

# Generate files.
make = render_file(p.make_jl, combined_view(p, t, pkg), tags(p))
precompile = render_file(p.precompile_jl, combined_view(p, t, pkg), tags(p))
gen_file(joinpath(build_dir, "make.jl"), make)
gen_file(joinpath(build_dir, "precompile.jl"), precompile)

# Create the compilation project.
with_project(build_dir) do
Pkg.add(PACKAGECOMPILER_DEP)
cd(() -> Pkg.develop(PackageSpec(; path="..")), build_dir)
end
end
30 changes: 30 additions & 0 deletions templates/build/make.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
using Pkg; Pkg.activate(@__DIR__)
using PackageCompiler

# List of packages to include in the sysimage
{{#SYSIMAGE_DEPS}}packages = Symbol.(keys(Pkg.project().dependencies)) # or packages = [:Plots, :DataFrames]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think we can use this because Pkg.project().dependencies is only defined in Julia v1.4+, and is not stable API anyway.

?Pkg.project says:

This feature requires Julia 1.4, and is considered experimental.

And we want to support as many v1.x versions as possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PkgDeps might be an alternative solution https://github.com/JuliaEcosystem/PkgDeps.jl (although the min Julia version there is 1.5)

{{/SYSIMAGE_DEPS}}
{{@SYSIMAGE_LIST}}packages = [
{{/SYSIMAGE_LIST}}
{{#SYSIMAGE_LIST}}
:{{{.}}},
{{/SYSIMAGE_LIST}}
{{@SYSIMAGE_LIST}}]
{{/SYSIMAGE_LIST}}

# Sysimage name
sysimage_name = "{{{SYSIMAGE_NAME}}}"

sysimage_ext = if Sys.iswindows()
".dll"
elseif Sys.isapple()
".dylib"
else
".so"
end

create_sysimage(
packages,
sysimage_path = sysimage_name * sysimage_ext,
precompile_execution_file = joinpath(@__DIR__, "precompile.jl"),
)
4 changes: 4 additions & 0 deletions templates/build/precompile.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
using {{{PKG}}}

# Put here any code that will trigger the compilation of as many methods as
# possible
69 changes: 69 additions & 0 deletions test/fixtures/AllPlugins/build/Manifest.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# This file is machine-generated - editing it directly is not advised

[[AllPlugins]]
path = ".."
uuid = "5b7e9947-ddc0-4b3f-9b55-0d8042f74170"
version = "0.1.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[LibGit2]]
deps = ["Printf"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[PackageCompiler]]
deps = ["Libdl", "Pkg", "UUIDs"]
git-tree-sha1 = "d448727c4b86be81b225b738c88d30334fda6779"
uuid = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
version = "1.2.5"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
3 changes: 3 additions & 0 deletions test/fixtures/AllPlugins/build/Project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[deps]
AllPlugins = "5b7e9947-ddc0-4b3f-9b55-0d8042f74170"
PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
22 changes: 22 additions & 0 deletions test/fixtures/AllPlugins/build/make.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
using Pkg; Pkg.activate(@__DIR__)
using PackageCompiler

# List of packages to include in the sysimage
packages = Symbol.(keys(Pkg.project().dependencies)) # or packages = [:Plots, :DataFrames]

# Sysimage name
sysimage_name = "sysimage"

sysimage_ext = if Sys.iswindows()
".dll"
elseif Sys.isapple()
".dylib"
else
".so"
end

create_sysimage(
packages,
sysimage_path = sysimage_name * sysimage_ext,
precompile_execution_file = joinpath(@__DIR__, "precompile.jl"),
)
4 changes: 4 additions & 0 deletions test/fixtures/AllPlugins/build/precompile.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
using AllPlugins

# Put here any code that will trigger the compilation of as many methods as
# possible
69 changes: 69 additions & 0 deletions test/fixtures/WackyOptions/build/Manifest.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# This file is machine-generated - editing it directly is not advised

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[LibGit2]]
deps = ["Printf"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[PackageCompiler]]
deps = ["Libdl", "Pkg", "UUIDs"]
git-tree-sha1 = "d448727c4b86be81b225b738c88d30334fda6779"
uuid = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
version = "1.2.5"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[WackyOptions]]
path = ".."
uuid = "5b7e9947-ddc0-4b3f-9b55-0d8042f74170"
version = "1.0.0"
3 changes: 3 additions & 0 deletions test/fixtures/WackyOptions/build/Project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[deps]
PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
WackyOptions = "5b7e9947-ddc0-4b3f-9b55-0d8042f74170"
25 changes: 25 additions & 0 deletions test/fixtures/WackyOptions/build/make.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
using Pkg; Pkg.activate(@__DIR__)
using PackageCompiler

# List of packages to include in the sysimage
packages = [
:Plots,
:DataFrames,
]

# Sysimage name
sysimage_name = "foo/bar/sysimage"

sysimage_ext = if Sys.iswindows()
".dll"
elseif Sys.isapple()
".dylib"
else
".so"
end

create_sysimage(
packages,
sysimage_path = sysimage_name * sysimage_ext,
precompile_execution_file = joinpath(@__DIR__, "precompile.jl"),
)
4 changes: 4 additions & 0 deletions test/fixtures/WackyOptions/build/precompile.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
using WackyOptions

# Put here any code that will trigger the compilation of as many methods as
# possible
7 changes: 6 additions & 1 deletion test/reference.jl
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ end
@testset "All plugins" begin
test_all("AllPlugins"; authors=USER, plugins=[
AppVeyor(), CirrusCI(), Citation(), Codecov(), CompatHelper(), Coveralls(),
Develop(), Documenter(), DroneCI(), GitHubActions(), GitLabCI(), TravisCI(), RegisterAction(),
Develop(), Documenter(), DroneCI(), GitHubActions(), GitLabCI(), PackageCompiler(),
RegisterAction(), TravisCI(),
])
end

Expand Down Expand Up @@ -126,6 +127,10 @@ end
GitHubActions(; x86=true, linux=false, coverage=false),
GitLabCI(; coverage=false, extra_versions=[v"0.6"]),
License(; name="ISC"),
PackageCompiler(;
sysimage_name=joinpath("foo", "bar", "sysimage"),
packages=[:Plots, "DataFrames"],
),
ProjectFile(; version=v"1"),
Readme(; inline_badges=true, badge_off=[Codecov]),
RegisterAction(; prompt="gimme version"),
Expand Down