Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scheduling improvements #425

Merged
merged 44 commits into from
Feb 14, 2025
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ef333b4
merge display methods
creativeprojects Oct 28, 2024
891081c
display only if more than one
creativeprojects Oct 28, 2024
e214faa
quick fix for #378
creativeprojects Oct 28, 2024
a3b4eef
keep trying to remove jobs
creativeprojects Oct 28, 2024
1160aa1
improve message on windows
creativeprojects Oct 28, 2024
faf1c5f
rename errors as schedule job
creativeprojects Oct 28, 2024
af9f96a
don't ask to start the job on darwin
creativeprojects Oct 28, 2024
7d8348b
improve launchd and systemd handlers
creativeprojects Oct 29, 2024
8658345
add cron entry parser
creativeprojects Oct 30, 2024
146173c
remove unused fields
creativeprojects Oct 31, 2024
b53dcb3
simplify line parsing
creativeprojects Oct 31, 2024
a82c6ea
add Scheduled method on systemd handler
creativeprojects Oct 31, 2024
3cc7fc7
run tests only on os supporting systemd
creativeprojects Oct 31, 2024
28b3109
wire up Scheduled method in crond handler
creativeprojects Oct 31, 2024
b3c4643
read env variables from systemd unit
creativeprojects Oct 31, 2024
8394100
read registered tasks
creativeprojects Nov 1, 2024
066d99f
fix interval test
creativeprojects Nov 1, 2024
2694b94
add scheduled for windows
creativeprojects Nov 1, 2024
f5044b5
add scheduled for windows
creativeprojects Nov 1, 2024
31358cb
use afero for crontab tests
creativeprojects Nov 1, 2024
73952eb
add config file to launchd Scheduled
creativeprojects Nov 1, 2024
c3a4a74
fix job tests on Windows
creativeprojects Nov 1, 2024
a0fa134
add tests for systemd
creativeprojects Nov 1, 2024
9f79f2a
add test on Read
creativeprojects Nov 1, 2024
d67f1c7
fix all tests on systemd handler
creativeprojects Nov 1, 2024
587490d
github windows agent is stupidely slow
creativeprojects Nov 1, 2024
5355cdc
refactoring launchd handler
creativeprojects Nov 2, 2024
7a097d1
add tests on crontab GetEntries
creativeprojects Nov 3, 2024
d51c9ba
remove empty Scheduler struct and call Handler directly
creativeprojects Nov 3, 2024
c11d105
removeJobs now searches for existing scheduled jobs to remove
creativeprojects Nov 3, 2024
53a9578
only display scheduled jobs in status command
creativeprojects Nov 4, 2024
3de6011
parse crontab line back into schedule event
creativeprojects Nov 4, 2024
c80e0c8
converts back launchd schedule into calendar event
creativeprojects Nov 4, 2024
201a3f8
improve display of launchd job status
creativeprojects Nov 4, 2024
3e80450
GA is completely useless
creativeprojects Nov 9, 2024
d3654ba
windows scheduler: add config file
creativeprojects Nov 13, 2024
0879084
fix SplitArgument to run with windows folders
creativeprojects Nov 13, 2024
240601a
fix unix tests under windows
creativeprojects Nov 13, 2024
e42c536
Merge branch 'master' into scheduling-improvements
creativeprojects Feb 4, 2025
8b41389
fix comments from AI code review
creativeprojects Feb 4, 2025
ac1ecef
AI nitpicking
creativeprojects Feb 4, 2025
44785cb
improve test coverage
creativeprojects Feb 4, 2025
623b5b2
added new integration tests & minor refactoring
creativeprojects Feb 5, 2025
003d987
more integration tests on create schedules
creativeprojects Feb 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ $(GOBIN)/mockery: verify $(GOBIN)/eget
@echo "[*] $@"
"$(GOBIN)/eget" vektra/mockery --upgrade-only --to '$(GOBIN)'

$(GOBIN)/golangci-lint: verify $(GOBIN)/eget
@echo "[*] $@"
"$(GOBIN)/eget" golangci/golangci-lint --tag v1.63.4 --asset=tar.gz --upgrade-only --to '$(GOBIN)'

prepare_build: verify download
@echo "[*] $@"

Expand Down Expand Up @@ -296,14 +300,14 @@ checklinks:
muffet -b 8192 --exclude="(linux.die.net|stackoverflow.com)" http://localhost:1313/resticprofile/

.PHONY: lint
lint:
lint: $(GOBIN)/golangci-lint
@echo "[*] $@"
GOOS=darwin golangci-lint run
GOOS=linux golangci-lint run
GOOS=windows golangci-lint run

.PHONY: fix
fix:
fix: $(GOBIN)/golangci-lint
@echo "[*] $@"
$(GOCMD) mod tidy
$(GOCMD) fix ./...
Expand Down
5 changes: 5 additions & 0 deletions calendar/event.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,11 @@ func (e *Event) Parse(input string) error {
func (e *Event) Next(from time.Time) time.Time {
// start from time and increment of 1 minute each time
next := from.Truncate(time.Minute) // truncate all the seconds
// if we're already partway through a minute, skip to the next minute
// to avoid scheduling events in the past
if from.Second() > 0 {
next = next.Add(time.Minute) // it's too late for the current minute
}
// should stop in 2 years time to avoid an infinite loop
endYear := from.Year() + 2
for next.Year() <= endYear {
Expand Down
27 changes: 17 additions & 10 deletions calendar/event_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -161,24 +161,31 @@ func TestNextTrigger(t *testing.T) {
// the base time is the example in the Go documentation https://golang.org/pkg/time/
ref, err := time.Parse(time.ANSIC, "Mon Jan 2 15:04:05 2006")
require.NoError(t, err)
refNoSecond, err := time.Parse(time.ANSIC, "Mon Jan 2 15:04:00 2006")
require.NoError(t, err)

testData := []struct{ event, trigger string }{
{"*:*:*", "2006-01-02 15:04:00"}, // seconds are zeroed out
{"03-*", "2006-03-01 00:00:00"},
{"*-01", "2006-02-01 00:00:00"},
{"*:*:11", "2006-01-02 15:04:00"}, // again, seconds are zeroed out
{"*:11:*", "2006-01-02 15:11:00"},
{"11:*:*", "2006-01-03 11:00:00"},
{"tue", "2006-01-03 00:00:00"},
{"2003-*-*", "0001-01-01 00:00:00"},
testData := []struct {
event, trigger string
ref time.Time
}{
{"*:*:*", "2006-01-02 15:04:00", refNoSecond}, // at the exact same second
{"*:*:*", "2006-01-02 15:05:00", ref}, // seconds are zeroed out => take next minute
{"03-*", "2006-03-01 00:00:00", ref},
{"*-01", "2006-02-01 00:00:00", ref},
{"*:*:11", "2006-01-02 15:04:00", refNoSecond}, // at the exact same second
{"*:*:11", "2006-01-02 15:05:00", ref}, // seconds are zeroed out => take next minute
{"*:11:*", "2006-01-02 15:11:00", ref},
{"11:*:*", "2006-01-03 11:00:00", ref},
{"tue", "2006-01-03 00:00:00", ref},
{"2003-*-*", "0001-01-01 00:00:00", ref},
}

for _, testItem := range testData {
t.Run(testItem.event, func(t *testing.T) {
event := NewEvent()
err = event.Parse(testItem.event)
assert.NoError(t, err)
assert.Equal(t, testItem.trigger, event.Next(ref).String()[0:len(testItem.trigger)])
assert.Equal(t, testItem.trigger, event.Next(testItem.ref).String()[0:len(testItem.trigger)])
})
}
}
Expand Down
1 change: 1 addition & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ ignore:
- run_profile.go
- syslog.go
- syslog_windows.go
- "**/mocks/*.go"

codecov:
notify:
Expand Down
12 changes: 9 additions & 3 deletions commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"slices"
"strconv"
"strings"
"sync"
"time"

"github.com/creativeprojects/clog"
Expand All @@ -25,6 +26,7 @@

var (
ownCommands = NewOwnCommands()
elevation sync.Once
)

func init() {
Expand Down Expand Up @@ -99,7 +101,8 @@
needConfiguration: true,
hide: false,
flags: map[string]string{
"--no-start": "don't start the timer/service (systemd/launch only)",
"--no-start": "don't start the job after installing (systemd/launch only)",
"--start": "start the job after installing (systemd/launch only)",
"--all": "add all scheduled jobs of all profiles and groups",
},
},
Expand Down Expand Up @@ -323,8 +326,11 @@
}
// maybe can find a better way than searching for the word "denied"?
if platform.IsWindows() && !flags.isChild && strings.Contains(err.Error(), "denied") {
clog.Info("restarting resticprofile in elevated mode...")
err := elevated()
// we try only once, otherwise we return the original error
elevation.Do(func() {
clog.Info("restarting resticprofile in elevated mode...")
err = elevated()
})

Check warning on line 333 in commands.go

View check run for this annotation

Codecov / codecov/patch

commands.go#L329-L333

Added lines #L329 - L333 were not covered by tests
if err != nil {
return err
}
Expand Down
129 changes: 84 additions & 45 deletions commands_schedule.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@
"golang.org/x/text/language"
)

const (
legacyFlagWarning = "the --legacy flag is only temporary and will be removed in version 1.0."
)

// createSchedule accepts one argument from the commandline: --no-start
func createSchedule(_ io.Writer, ctx commandContext) error {
c := ctx.config
Expand All @@ -24,9 +28,9 @@
defer c.DisplayConfigurationIssues()

type profileJobs struct {
scheduler schedule.SchedulerConfig
name string
jobs []*config.Schedule
schedulerConfig schedule.SchedulerConfig
name string
jobs []*config.Schedule
}

allJobs := make([]profileJobs, 0, 1)
Expand Down Expand Up @@ -55,12 +59,12 @@
}
}

allJobs = append(allJobs, profileJobs{scheduler: scheduler, name: profileName, jobs: jobs})
allJobs = append(allJobs, profileJobs{schedulerConfig: scheduler, name: profileName, jobs: jobs})

Check warning on line 62 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L62

Added line #L62 was not covered by tests
}

// Step 2: Schedule all collected jobs
for _, j := range allJobs {
err := scheduleJobs(schedule.NewHandler(j.scheduler), j.name, j.jobs)
err := scheduleJobs(schedule.NewHandler(j.schedulerConfig), j.jobs)

Check warning on line 67 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L67

Added line #L67 was not covered by tests
if err != nil {
return retryElevated(err, flags)
}
Expand All @@ -70,26 +74,45 @@
}

func removeSchedule(_ io.Writer, ctx commandContext) error {
var err error

Check warning on line 77 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L77

Added line #L77 was not covered by tests
c := ctx.config
flags := ctx.flags
args := ctx.request.arguments

// Unschedule all jobs of all selected profiles
for _, profileName := range selectProfilesAndGroups(c, flags, args) {
profileFlags := flagsForProfile(flags, profileName)
if slices.Contains(args, "--legacy") {
clog.Warning(legacyFlagWarning)
// Unschedule all jobs of all selected profiles
for _, profileName := range selectProfilesAndGroups(c, flags, args) {
profileFlags := flagsForProfile(flags, profileName)

Check warning on line 86 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L82-L86

Added lines #L82 - L86 were not covered by tests

scheduler, jobs, err := getRemovableScheduleJobs(c, profileFlags)
if err != nil {
return err
}
schedulerConfig, jobs, err := getRemovableScheduleJobs(c, profileFlags)
if err != nil {
return err
}

Check warning on line 91 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L88-L91

Added lines #L88 - L91 were not covered by tests

err = removeJobs(schedule.NewHandler(scheduler), profileName, jobs)
if err != nil {
return retryElevated(err, flags)
err = removeJobs(schedule.NewHandler(schedulerConfig), jobs)
if err != nil {
err = retryElevated(err, flags)
}
if err != nil {
// we keep trying to remove the other jobs
clog.Error(err)
}

Check warning on line 100 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L93-L100

Added lines #L93 - L100 were not covered by tests
}
return nil

Check warning on line 102 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L102

Added line #L102 was not covered by tests
}

return nil
profileName := ctx.request.profile
if slices.Contains(args, "--all") {
// Unschedule all jobs of all profiles
profileName = ""
}
schedulerConfig := schedule.NewSchedulerConfig(ctx.global)
err = removeScheduledJobs(schedule.NewHandler(schedulerConfig), ctx.config.GetConfigFile(), profileName)
if err != nil {
err = retryElevated(err, flags)
}
return err

Check warning on line 115 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L105-L115

Added lines #L105 - L115 were not covered by tests
}

func statusSchedule(w io.Writer, ctx commandContext) error {
Expand All @@ -99,38 +122,54 @@

defer c.DisplayConfigurationIssues()

if !slices.Contains(args, "--all") {
scheduler, schedules, _, err := getScheduleJobs(c, flags)
if err != nil {
return err
}
if len(schedules) == 0 {
clog.Warningf("profile or group %s has no schedule", flags.name)
if slices.Contains(flags.resticArgs, "--legacy") {
clog.Warning(legacyFlagWarning)
// single profile or group
if !slices.Contains(args, "--all") {
schedulerConfig, schedules, _, err := getScheduleJobs(c, flags)
if err != nil {
return err
}
if len(schedules) == 0 {
clog.Warningf("profile or group %s has no schedule", flags.name)
return nil
}
err = statusScheduleProfileOrGroup(schedulerConfig, schedules, flags)
if err != nil {
return err
}

Check warning on line 140 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L126-L140

Added lines #L126 - L140 were not covered by tests
return nil
}
err = statusScheduleProfileOrGroup(scheduler, schedules, flags)
if err != nil {
return err
}
}

for _, profileName := range selectProfilesAndGroups(c, flags, args) {
profileFlags := flagsForProfile(flags, profileName)
scheduler, schedules, schedulable, err := getScheduleJobs(c, profileFlags)
if err != nil {
return err
}
// it's all fine if this profile has no schedule
if len(schedules) == 0 {
continue
}
clog.Infof("%s %q:", cases.Title(language.English).String(schedulable.Kind()), profileName)
err = statusScheduleProfileOrGroup(scheduler, schedules, profileFlags)
if err != nil {
// display the error but keep going with the other profiles
clog.Error(err)
// all profiles and groups
for _, profileName := range selectProfilesAndGroups(c, flags, args) {
profileFlags := flagsForProfile(flags, profileName)
scheduler, schedules, schedulable, err := getScheduleJobs(c, profileFlags)
if err != nil {
return err
}

Check warning on line 150 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L145-L150

Added lines #L145 - L150 were not covered by tests
// it's all fine if this profile has no schedule
if len(schedules) == 0 {
continue

Check warning on line 153 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L152-L153

Added lines #L152 - L153 were not covered by tests
}
clog.Infof("%s %q:", cases.Title(language.English).String(schedulable.Kind()), profileName)
err = statusScheduleProfileOrGroup(scheduler, schedules, profileFlags)
if err != nil {
// display the error but keep going with the other profiles
clog.Error(err)
}

Check warning on line 160 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L155-L160

Added lines #L155 - L160 were not covered by tests
}
}
profileName := ctx.request.profile
if slices.Contains(args, "--all") {
// display all jobs of all profiles
profileName = ""
}

Check warning on line 167 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L165-L167

Added lines #L165 - L167 were not covered by tests
schedulerConfig := schedule.NewSchedulerConfig(ctx.global)
err := statusScheduledJobs(schedule.NewHandler(schedulerConfig), ctx.config.GetConfigFile(), profileName)
if err != nil {
return retryElevated(err, flags)
}

Check warning on line 172 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L171-L172

Added lines #L171 - L172 were not covered by tests
return nil
}

Expand Down Expand Up @@ -159,8 +198,8 @@
return flags
}

func statusScheduleProfileOrGroup(scheduler schedule.SchedulerConfig, schedules []*config.Schedule, flags commandLineFlags) error {
err := statusJobs(schedule.NewHandler(scheduler), flags.name, schedules)
func statusScheduleProfileOrGroup(schedulerConfig schedule.SchedulerConfig, schedules []*config.Schedule, flags commandLineFlags) error {
err := statusJobs(schedule.NewHandler(schedulerConfig), flags.name, schedules)

Check warning on line 202 in commands_schedule.go

View check run for this annotation

Codecov / codecov/patch

commands_schedule.go#L201-L202

Added lines #L201 - L202 were not covered by tests
if err != nil {
return retryElevated(err, flags)
}
Expand Down
28 changes: 22 additions & 6 deletions commands_schedule_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,40 +37,55 @@ profiles:

profile-schedule-inline:
backup:
schedule: daily
schedule: "*:00,30"

profile-schedule-struct:
backup:
schedule:
at: daily
at: "*:00,30"

`

func TestCommandsIntegrationUsingCrontab(t *testing.T) {
const scheduleIntegrationTestsCrontab = `
### this content was generated by resticprofile, please leave this line intact ###
00,30 * * * * user cd /workdir && /home/resticprofile --no-ansi --config config.yaml run-schedule backup@profile
### end of resticprofile content, please leave this line intact ###
`
creativeprojects marked this conversation as resolved.
Show resolved Hide resolved

// TODO: finish implementation of this test
func TestStatusCommandIntegrationUsingCrontab(t *testing.T) {
crontab := filepath.Join(t.TempDir(), "crontab")
err := os.WriteFile(crontab, []byte(scheduleIntegrationTestsCrontab), 0o600)
require.NoError(t, err)

cfg, err := config.Load(
bytes.NewBufferString(fmt.Sprintf(scheduleIntegrationTestsConfiguration, crontab)),
config.FormatYAML,
config.WithConfigFile("config.yaml"),
)
require.NoError(t, err)
require.NotNil(t, cfg)

global, err := cfg.GetGlobalSection()
require.NoError(t, err)
require.NotNil(t, global)

testCases := []struct {
name string
contains string
err error
}{
{
name: "",
err: config.ErrNotFound,
err: nil,
},
{
name: "profile-schedule-inline",
contains: "Original form: daily",
contains: "Original form: *-*-* *:00,30:00",
},
{
name: "profile-schedule-struct",
contains: "Original form: daily",
contains: "Original form: *-*-* *:00,30:00",
},
}

Expand All @@ -82,6 +97,7 @@ func TestCommandsIntegrationUsingCrontab(t *testing.T) {
flags: commandLineFlags{
name: tc.name,
},
global: global,
},
}
output := &bytes.Buffer{}
Expand Down
Loading