Integrate black forest labs FLUX.1 models into Galaxy (#1496)

* Integrate black forest labs FLUX.1 models into Galaxy * using loc files * Update diffusers package version to 0.30.2 + replace space with tab in loc file * update loc file * Update * Update FLUX tool to support both file and text-based prompts * init tests and help section * add readme for user preferences * Fix errors * using huggingface_hub to avoid using env variable that will cause imports to not be at the beginning of the file * Update huggingface.loc.sample with pipeline tags, domain, free_tag, and version * Update huggingface.loc.sample with corrected model tags and version * Fix test + removing huggingface_hub_token * Remove unnecessary links and references in flux.xml * Refactor flux.py to remove unnecessary code and improve performance * Update tools/flux/tool-data/huggingface.loc.sample * add tool_data_table_conf.xml.sample file --------- Co-authored-by: Björn Grüning <[email protected]>
bgruening · Oct 14, 2024 · 71b3dac · 71b3dac
1 parent f42263c
commit 71b3dac
Show file tree

Hide file tree

Showing 6 changed files with 172 additions and 0 deletions.
diff --git a/tools/flux/.shed.yml b/tools/flux/.shed.yml
@@ -0,0 +1,14 @@
+name: black_forest_labs_flux
+owner: bgruening
+description: Using the black forest labs FLUX.1 models to generate image based on user input.
+long_description: |
+  This tool uses the black forest labs FLUX.1 models to generate image based on user input.
+  Users can specify the model (dev,schnell),and provide prompt.
+  The tool will then generate and return the corresponding image based on the input provided.
+remote_repository_url: https://github.com/bgruening/galaxytools/tree/master/tools/flux
+homepage_url: https://github.com/bgruening/galaxytools/tree/master/tools/cflux
+type:
+categories:
+  - Machine Learning
+maintainers:
+  - arash77
diff --git a/tools/flux/flux.py b/tools/flux/flux.py
@@ -0,0 +1,32 @@
+import sys
+
+import torch
+from diffusers import FluxPipeline
+
+model = sys.argv[1]
+
+prompt_type = sys.argv[2]
+if prompt_type == "file":
+    with open(sys.argv[3], "r") as f:
+        prompt = f.read().strip()
+elif prompt_type == "text":
+    prompt = sys.argv[3]
+
+if model not in ["black-forest-labs/FLUX.1-dev", "black-forest-labs/FLUX.1-schnell"]:
+    print("Invalid model!")
+    sys.exit(1)
+
+
+pipe = FluxPipeline.from_pretrained(model, torch_dtype=torch.bfloat16)
+pipe.enable_sequential_cpu_offload()
+pipe.vae.enable_slicing()
+pipe.vae.enable_tiling()
+pipe.to(torch.float16)
+
+image = pipe(
+    prompt,
+    num_inference_steps=4,
+    generator=torch.Generator("cpu").manual_seed(42),
+).images[0]
+
+image.save("output.png")
diff --git a/tools/flux/flux.xml b/tools/flux/flux.xml
@@ -0,0 +1,100 @@
+<tool id="black_forest_labs_flux" name="FLUX" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="23.0">
+    <description>text-to-image model</description>
+    <macros>
+        <token name="@TOOL_VERSION@">2024</token>
+        <token name="@VERSION_SUFFIX@">0</token>
+    </macros>
+    <requirements>
+        <requirement type="package" version="3.12">python</requirement>
+        <requirement type="package" version="2.4.1">pytorch</requirement>
+        <requirement type="package" version="0.19.1">torchvision</requirement>
+        <requirement type="package" version="0.30.2">diffusers</requirement>
+        <requirement type="package" version="4.44.2">transformers</requirement>
+        <requirement type="package" version="0.34.0">accelerate</requirement>
+        <requirement type="package" version="0.2.0">sentencepiece</requirement>
+        <requirement type="package" version="4.25.3">protobuf</requirement>
+        <requirement type="package" version="0.24.6">huggingface_hub</requirement>
+    </requirements>
+    <command detect_errors="exit_code"><![CDATA[
+export HF_HOME='$flux_models.fields.path' &&
+python '$__tool_directory__/flux.py'
+'$flux_models'
+'$input_type_selector'
+'$prompt'
+    ]]></command>
+    <configfiles>
+    </configfiles>
+    <inputs>
+        <param name="flux_models" label="Model data" type="select" help="contact the administrator of this Galaxy instance if you miss model data">
+            <options from_data_table="huggingface">
+                <filter type="static_value" column="4" value="flux"/>
+                <filter type="static_value" column="5" value="1"/>
+            </options>
+            <validator message="No model annotation is available for FLUX" type="no_options"/>
+        </param>
+        <conditional name="input_type">
+            <param name="input_type_selector" type="select" label="Choose the type of input">
+                <option value="file" selected="true">File based input</option>
+                <option value="text">Text based input</option>
+            </param>
+            <when value="file">
+                <param name="prompt" type="data" optional="false" format="txt" label="Prompt file" help="This data will be used as prompt"/>
+            </when>
+            <when value="text">
+                <param name="prompt" type="text" optional="false" label="Prompt" help="This text will be used as prompt" area="true">
+                    <validator type="empty_field"/>
+                </param>
+            </when>
+        </conditional>
+    </inputs>
+    <outputs>
+        <data name="output" format="png" label="${tool.name} on ${on_string}" from_work_dir="./output.png"/>
+    </outputs>
+    <tests>
+        <test expect_exit_code="1" expect_failure="true">
+            <param name="flux_models" value="unknown"/>
+            <param name="input_type_selector" value="file"/>
+            <param name="prompt" value="flux_test.txt"/>
+            <assert_stdout>
+                <has_text text="Invalid model!"/>
+            </assert_stdout>
+        </test>
+    </tests>
+    <help><![CDATA[
+
+.. class:: infomark
+
+**What it does**
+
+FLUX is a text-to-image model that generates images from textual descriptions.
+
+Usage
+.....
+
+**Input**
+
+1. **Model data**:
+    Select the model data from the dropdown list. The available models are dev and schnell.
+    For using dev model, make sure to not use it for commercial purposes.
+
+2. **Input Prompt**: You can provide the input prompt in two ways:
+    - **File based input**: Upload a txt file containing the prompt.
+    - **Text based input**: Enter the prompt text in the text area.
+
+**Output**
+
+This response is saved in the `output.png` file.
+
+    ]]></help>
+    <citations>
+        <citation type="bibtex">
+            @misc{flux,
+                author = {black forest labs},
+                title = {FLUX github repository},
+                howpublished = {\url{https://github.com/black-forest-labs/flux}},
+                year = {2024},
+                note = {Accessed: 2024-09-04}
+            }
+        </citation>
+    </citations>
+</tool>
diff --git a/tools/flux/test-data/flux_test.txt b/tools/flux/test-data/flux_test.txt
@@ -0,0 +1 @@
+Extreme close-up of a spiral galaxy, direct frontal view, with highly detailed stars and nebulae. The image has sharp focus on texture and color, with ethereal cosmic lighting to capture the galaxy's natural shine and depth. The words 'Galaxy Project' are painted in large, white brush strokes with visible texture, centered across the image.
diff --git a/tools/flux/tool-data/huggingface.loc.sample b/tools/flux/tool-data/huggingface.loc.sample
@@ -0,0 +1,18 @@
+
+#This is a sample file distributed with Galaxy that is used to define huggingface
+#models, using 7 columns tab separated
+#(longer whitespace are TAB characters):
+#
+#The entries are as follows:
+#
+#<unique_build_id>	<display_name>	<pipeline_tag>	<domain>	<free_tag>	<version>	<folder_base_path>
+#
+#pipeline_tag (see this URL: https://huggingface.co/models?pipeline_tag=text-to-image they call it that way)
+#domain (defined by the Galaxy community, e.g. image / sequence / text)
+#free_tag (not idea yet, but could be freely used by an admin to specify more filter options)
+#version (if available)
+#Your huggingface.loc file should include the huggingface cached model folder you have stored.
+#For example:
+#
+#black-forest-labs/FLUX.1-dev	FLUX.1 [dev] is an open-weight, guidance-distilled model for non-commercial applications.	text-to-image	image	flux	1	/path/to/huggingface/that-contains/hub/models--black-forest-labs--FLUX.1-dev
+#black-forest-labs/FLUX.1-schnell	FLUX.1 [schnell] is the fastest model, tailored for personal use.	text-to-image	image	flux	1	/path/to/huggingface/that-contains/hub/models--black-forest-labs--FLUX.1-schnell
diff --git a/tools/flux/tool_data_table_conf.xml.sample b/tools/flux/tool_data_table_conf.xml.sample
@@ -0,0 +1,7 @@
+<!-- Paths are relative to the value of `tool_data_path` in galaxy.ini -->
+<tables>
+    <table name="huggingface" comment_char="#" allow_duplicate_entries="False">
+        <columns>value, name, pipeline_tag, domain, free_tag, version, path</columns>
+        <file path="huggingface.loc" />
+    </table>
+</tables>
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Extreme close-up of a spiral galaxy, direct frontal view, with highly detailed stars and nebulae. The image has sharp focus on texture and color, with ethereal cosmic lighting to capture the galaxy's natural shine and depth. The words 'Galaxy Project' are painted in large, white brush strokes with visible texture, centered across the image.