Skip to content

Commit

Permalink
removing 18fgsa references and removing unused testing pipeline
Browse files Browse the repository at this point in the history
  • Loading branch information
dandersonsw committed Nov 8, 2024
1 parent c4b02d7 commit 053ef57
Show file tree
Hide file tree
Showing 7 changed files with 41 additions and 59 deletions.
9 changes: 4 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,22 @@ We want to ensure a welcoming environment for all of our projects. Our staff fol

We encourage you to read this project's CONTRIBUTING policy (you are here), its [LICENSE](LICENSE.md), and its [README](README.md).

If you have any questions or want to read more, check out the [18F Open Source Policy GitHub repository]( https://github.com/18f/open-source-policy), or just [shoot us an email](mailto:[email protected]).
If you have any questions or want to read more, check out the [18F Open Source Policy GitHub repository](https://github.com/18f/open-source-policy), or just [shoot us an email](mailto:[email protected]).

## Development

Requires [Docker](https://www.docker.com/).

1. Run `cp config.example.json config.json`.
1. Modify `config.json`.
* See [the instructions for getting your AWS credentials](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html#cli-signup).
* Exclude the `s3://` prefix/protocol for `bucket`.

- See [the instructions for getting your AWS credentials](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html#cli-signup).
- Exclude the `s3://` prefix/protocol for `bucket`.

1. Run `./test/out </full/path/to/dir>`.
1. Run `./test/in </full/path/to/dir>`.
1. Run `./test/check`.

Every tag and branch created on this repository is automatically built on [Docker Hub](https://hub.docker.com/r/18fgsa/s3-resource-simple/).

## Public domain

This project is in the public domain within the United States, and
Expand Down
56 changes: 33 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,40 @@

Resource to upload files to S3. Unlike the [the official S3 Resource](https://github.com/concourse/s3-resource), this Resource can upload or download multiple files.

## Usage
## Usage

Include the following in your Pipeline YAML file, replacing the values in the angle brackets (`< >`):

```yaml
resource_types:
- name: <resource type name>
type: docker-image
source:
repository: 18fgsa/s3-resource-simple
- name: <resource type name>
type: registry-image
source:
aws_access_key_id: ((ecr_aws_key))
aws_secret_access_key: ((ecr_aws_secret))
repository: s3-resource-simple
aws_region: ((aws_region))
tag: ((tag))
resources:
- name: <resource name>
type: <resource type name>
source:
access_key_id: {{aws-access-key}}
secret_access_key: {{aws-secret-key}}
bucket: {{aws-bucket}}
path: [<optional>, use to sync to a specific path of the bucket instead of root of bucket]
change_dir_to: [<optional, see note below>]
options: [<optional, see note below>]
region: <optional, see below>
- name: <resource name>
type: <resource type name>
source:
access_key_id: { { aws-access-key } }
secret_access_key: { { aws-secret-key } }
bucket: { { aws-bucket } }
path:
[
<optional>,
use to sync to a specific path of the bucket instead of root of bucket,
]
change_dir_to: [<optional, see note below>]
options: [<optional, see note below>]
region: <optional, see below>
jobs:
- name: <job name>
plan:
- <some Resource or Task that outputs files>
- put: <resource name>
- name: <job name>
plan:
- <some Resource or Task that outputs files>
- put: <resource name>
```
## AWS Credentials
Expand All @@ -44,7 +52,9 @@ test
├── 1.json
└── 2.json
```

and the config:

```
- name: test
type: s3-resource-simple
Expand Down Expand Up @@ -76,13 +86,13 @@ we can upload _only_ the `results` subdirectory by using the following `options`

```yaml
options:
- "--exclude '*'"
- "--include 'results/*'"
- "--exclude '*'"
- "--include 'results/*'"
```

### Region
Interacting with some AWS regions (like London) requires AWS Signature Version
4. This options allows you to explicitly specify region where your bucket is

Interacting with some AWS regions (like London) requires AWS Signature Version 4. This options allows you to explicitly specify region where your bucket is
located (if this is set, AWS_DEFAULT_REGION env variable will be set accordingly).

```yaml
Expand Down
2 changes: 1 addition & 1 deletion test/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

set -e

docker build -t 18fgsa/s3-resource-simple .
docker build -t s3-resource-simple .
2 changes: 1 addition & 1 deletion test/check
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ source "$(dirname $0)/build.sh"
echo $json | docker run \
-i \
--rm \
18fgsa/s3-resource-simple \
s3-resource-simple \
/opt/resource/check
2 changes: 1 addition & 1 deletion test/in
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ echo $json | docker run \
-i \
--rm \
-v $dest:/tmp/output \
18fgsa/s3-resource-simple \
s3-resource-simple \
/opt/resource/in /tmp/output
2 changes: 1 addition & 1 deletion test/out
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ echo $json | docker run \
-i \
--rm \
-v $source:/tmp/input \
18fgsa/s3-resource-simple \
s3-resource-simple \
/opt/resource/out /tmp/input
27 changes: 0 additions & 27 deletions test/pipeline.yml

This file was deleted.

0 comments on commit 053ef57

Please sign in to comment.