Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Update docs #762

Merged
merged 19 commits into from
Jan 1, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
name: Release

on:
create:
branches:
- main
push:
tags:
- "v[0-9]+.[0-9]+.[0-9]+"
- "v[0-9]+.[0-9]+.[0-9]+-*"
Expand Down
4 changes: 3 additions & 1 deletion .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,10 @@ linters-settings:
values:
const:
AUTHOR: Yota Hamada
regexp:
YEAR: 20\d\d
template: |-
Copyright (C) {{ YEAR }} {{ AUTHOR }}
Copyright (C) {{YEAR}} {{ AUTHOR }}
SPDX-License-Identifier: GPL-3.0-or-later

run:
Expand Down
158 changes: 48 additions & 110 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,109 +26,53 @@

<h1><b>Dagu</b></h1>

Dagu is a powerful Cron alternative that comes with a Web UI. It allows you to define dependencies between commands as a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph) in a declarative [YAML format](https://dagu.readthedocs.io/en/latest/yaml_format.html). Dagu simplifies the management and execution of complex workflows. It natively supports running Docker containers, making HTTP requests, and executing commands over SSH.

- **Check out our** [Documentation](https://dagu.readthedocs.io)
- **Join our** [Discord Community](https://discord.gg/gpahPUjGRk)
- **Explore** [Quick Start Guide](https://dagu.readthedocs.io/en/latest/quickstart.html)
A powerful, self-contained Cron alternative with a clean Web UI and a [declarative YAML-based workflow definition](https://dagu.readthedocs.io/en/latest/yaml_format.html). Dagu simplifies complex job dependencies and scheduling with minimal overhead.

---

## **Highlights**

- Single binary file installation
- Declarative YAML format for defining DAGs
- Web UI for visually managing, rerunning, and monitoring pipelines
- Use existing programs without any modification
- Self-contained, with no need for a DBMS

## **Table of Contents**

- [**Highlights**](#highlights)
- [**Table of Contents**](#table-of-contents)
- [**Features**](#features)
- [**Use Cases**](#use-cases)
- [**Web UI**](#web-ui)
- [DAG Details](#dag-details)
- [DAGs](#dags)
- [Search](#search)
- [Execution History](#execution-history)
- [Log Viewer](#log-viewer)
- [**Installation**](#installation)
- [Via Bash script](#via-bash-script)
- [Via GitHub Releases Page](#via-github-releases-page)
- [Via Homebrew (macOS)](#via-homebrew-macos)
- [Via Docker](#via-docker)
- [**Quick Start Guide**](#quick-start-guide)
- [1. Launch the Web UI](#1-launch-the-web-ui)
- [2. Create a New DAG](#2-create-a-new-dag)
- [3. Edit the DAG](#3-edit-the-dag)
- [4. Execute the DAG](#4-execute-the-dag)
- [**CLI**](#cli)
- [**Remote Node Management support**](#remote-node-management-support)
- [Configuration](#configuration)
- [**Localized Documentation**](#localized-documentation)
- [**Documentation**](#documentation)
- [**Example DAG**](#example-dag)
- [Minimum examples](#minimum-examples)
- [Complex example](#complex-example)
- [**Running as a daemon**](#running-as-a-daemon)
- [**Motivation**](#motivation)
- [**Why Not Use an Existing DAG Scheduler Like Airflow?**](#why-not-use-an-existing-dag-scheduler-like-airflow)
- [**How It Works**](#how-it-works)
- [**License**](#license)
- [**Support and Community**](#support-and-community)
- **Simple Installation**: Single binary, no dependencies
- **Intuitive Web UI**: Visualize, monitor, and control workflows
- **YAML-based**: Define workflows in simple YAML files
- **Built-in Executors**: Support for Docker, HTTP, SSH, and more
- **Zero Config**: No database required, works out of the box

## **Features**

- Web User Interface
- Command Line Interface (CLI) with several commands for running and managing DAGs
- YAML format for defining DAGs, with support for various features including:
- Execution of custom code snippets
- Parameters
- Command substitution
- Conditional logic
- Web UI & CLI
- Web API Interface
- Powerful DAG definition in YAML format:
- Code snippets, parameters, environment variables
- Command substitution, piping, conditional logic
- Redirection of stdout and stderr
- Lifecycle hooks
- Repeating task
- Automatic retry
- Executors for running different types of tasks:
- Running arbitrary Docker containers
- Making HTTP requests
- Sending emails
- Running jq command
- Executing remote commands via SSH
- Remote Node support for managing multiple Dagu instances:
- Monitor DAGs across different environments
- Switch between nodes through UI dropdown
- Centralized management interface
- Lifecycle hooks (on failure, on exit, etc.)
- Repeating tasks, automatic/manual retry
- Run sub-DAGs
- Handy built-in executors:
- Docker containers
- HTTP requests
- Email sending
- JSON query with jq
- SSH remote commands
- Remote Dagu node management
- Email notification
- Scheduling with Cron expressions
- REST API Interface
- Basic Authentication over HTTPS

## **Use Cases**
## **Community**

- **Data Pipeline Automation**: Schedule ETL tasks for data processing
- **Infrastructure Monitoring**: Periodic health checks via HTTP or SSH
- **Automated Reporting**: Generate and send routine email reports
- **Batch Processing**: Automate data cleansing, model training, etc.
- **Task Dependency Management**: Handle interdependent processes seamlessly
- **Microservices Orchestration**: Manage and monitor microservice dependencies
- **CI/CD Integration**: Automate code deployment and testing
- **Alerting System**: Trigger notifications upon specific conditions
- **Custom Task Automation**: Run arbitrary scripts or commands with ease
- Issues: [GitHub Issues](https://github.com/dagu-org/dagu/issues)
- Discussion: [GitHub Discussions](https://github.com/dagu-org/dagu/discussions)
- Chat: [Discord](https://discord.gg/gpahPUjGRk)

## **Web UI**

### DAG Details

Real-time statuses, logs, and configuration details for each DAG. Easily edit configurations in your browser.
Real-time status, logs, and configuration for each DAG. Toggle graph orientation from the top-right corner.

![example](assets/images/demo.gif?raw=true)

Switch graph orientation with the toggle button at the top-right corner:

![Details-TD](assets/images/ui-details2.webp?raw=true)

### DAGs
Expand Down Expand Up @@ -189,11 +133,11 @@ docker run \
-p 8080:8080 \
-v $HOME/.config/dagu/dags:/home/dagu/.config/dagu/dags \
-v $HOME/.local/share/dagu:/home/dagu/.local/share/dagu \
-e DAGU_TZ=Asia/Tokyo \
-e DAGU_TZ=`ls -l /etc/localtime | awk -F'/zoneinfo/' '{print $2}'` \
ghcr.io/dagu-org/dagu:latest dagu start-all
```

Note: The environment variable `DAGU_TZ` is the timezone for the scheduler and server. You can set it to your local timezone.
Note: The environment variable `DAGU_TZ` is the timezone for the scheduler and server. You can set it to your local timezone (e.g. `America/New_York`).

See [Environment variables](https://dagu.readthedocs.io/en/latest/config.html#environment-variables) to configure those default directories.

Expand All @@ -217,13 +161,15 @@ Example:

```yaml
schedule: "* * * * *" # Run the DAG every minute
params:
- NAME: "Dagu"
steps:
- name: s1
command: echo Hello Dagu
- name: s2
command: echo done!
- name: Hello world
command: echo Hello $NAME
- name: Done
command: echo Done!
depends:
- s1
- Hello world
```

### 4. Execute the DAG
Expand Down Expand Up @@ -264,7 +210,7 @@ dagu scheduler [--dags=<path to directory>]
dagu version
```

## **Remote Node Management support**
## **Remote Node Management**

Dagu supports managing multiple Dagu servers from a single UI through its remote node feature. This allows you to:

Expand All @@ -276,22 +222,16 @@ See [Remote Node Configuration](https://dagu.readthedocs.io/en/latest/config_rem

### Configuration

Remote nodes can be configured by creating `admin.yaml` in `$HOME/.config/dagu/`:
Create `admin.yaml` in `$HOME/.config/dagu/`:

```yaml
# admin.yaml
remoteNodes:
- name: "prod" # Name of the remote node
apiBaseUrl: "https://prod.example.com/api/v1" # Base URL of the remote node API
- name: "prod"
apiBaseUrl: "https://prod.example.com/api/v1"
- name: "staging"
apiBaseUrl: "https://staging.example.com/api/v1"
```

## **Localized Documentation**

- [中文文档 (Chinese Documentation)](https://dagu.readthedocs.io/zh)
- [日本語ドキュメント (Japanese Documentation)](https://dagu.readthedocs.io/ja)

## **Documentation**

- [Installation Instructions](https://dagu.readthedocs.io/en/latest/installation.html)
Expand Down Expand Up @@ -329,18 +269,20 @@ remoteNodes:

## **Example DAG**

### Minimum examples
### Minimal DAG Definition

A DAG with two steps:

```yaml
params:
- NAME: "Dagu"
steps:
- name: step 1
command: echo hello
- name: step 2
command: echo world
- name: Hello world
command: echo Hello $NAME
- name: Done
command: echo Done!
depends:
- step 1
- Hello world
```

Using a pipe:
Expand All @@ -362,7 +304,7 @@ steps:

Note: The default shell is `$SHELL` or `sh`.

### Complex example
### Complex DAG Definition

A typical data pipeline for DevOps/Data Engineering scenarios:

Expand Down Expand Up @@ -483,7 +425,3 @@ We welcome any and all contributions!
## **License**

This project is licensed under the GNU GPLv3.

## **Support and Community**

Join our [Discord community](https://discord.gg/gpahPUjGRk) to ask questions, request features, and share your ideas.
63 changes: 63 additions & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,66 @@
build
.rye/
rye.lock

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
venv/
ENV/
env.bak/
venv.bak/
*.egg-info/
.eggs/
dist/
*.egg
*.whl

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Sphinx documentation
docs/_build/
docs/build/

# Local virtual environments
.venv/
7 changes: 3 additions & 4 deletions docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,6 @@ install-libs:
build:
sphinx-build -b html -D language=ja_JP source/ build/html

gettext-ja:
sphinx-build -b gettext source/ build/html
sphinx-intl update -p _build/gettext -l ja

PYTHON_VER = 3.7.16

venv:
Expand All @@ -43,3 +39,6 @@ venv:

install-python:
@pyenv install $(PYTHON_VER)

gen-pot-files:
sphinx-intl update -p build/gettext -l zh -l ja
Loading
Loading