Skip to content

Commit

Permalink
Bump Spark to 3.5.2
Browse files Browse the repository at this point in the history
  • Loading branch information
msmygit committed Aug 26, 2024
1 parent 012fd1f commit 34e0f11
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 7 deletions.
8 changes: 4 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ RUN mkdir -p /assets/ && cd /assets && \
curl -OL https://downloads.datastax.com/enterprise/cqlsh-astra.tar.gz && \
tar -xzf ./cqlsh-astra.tar.gz && \
rm ./cqlsh-astra.tar.gz && \
curl -OL https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz && \
tar -xzf ./spark-3.5.1-bin-hadoop3-scala2.13.tgz && \
rm ./spark-3.5.1-bin-hadoop3-scala2.13.tgz
curl -OL https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3-scala2.13.tgz && \
tar -xzf ./spark-3.5.2-bin-hadoop3-scala2.13.tgz && \
rm ./spark-3.5.2-bin-hadoop3-scala2.13.tgz

RUN apt-get update && apt-get install -y openssh-server vim python3 --no-install-recommends && \
rm -rf /var/lib/apt/lists/* && \
Expand Down Expand Up @@ -44,7 +44,7 @@ RUN chmod +x ./get-latest-maven-version.sh && \
rm -rf "$USER_HOME_DIR/.m2"

# Add all migration tools to path
ENV PATH="${PATH}:/assets/dsbulk/bin/:/assets/cqlsh-astra/bin/:/assets/spark-3.5.1-bin-hadoop3-scala2.13/bin/"
ENV PATH="${PATH}:/assets/dsbulk/bin/:/assets/cqlsh-astra/bin/:/assets/spark-3.5.2-bin-hadoop3-scala2.13/bin/"

EXPOSE 22

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ Migrate and Validate Tables between Origin and Target Cassandra Clusters.

### Prerequisite
- Install **Java11** (minimum) as Spark binaries are compiled with it.
- Install Spark version [`3.5.1`](https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz) on a single VM (no cluster necessary) where you want to run this job. Spark can be installed by running the following: -
- Install Spark version [`3.5.2`](https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3-scala2.13.tgz) on a single VM (no cluster necessary) where you want to run this job. Spark can be installed by running the following: -
```
wget https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz
tar -xvzf spark-3.5.1-bin-hadoop3-scala2.13.tgz
wget https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3-scala2.13.tgz
tar -xvzf spark-3.5.2-bin-hadoop3-scala2.13.tgz
```

> :warning: If the above Spark and Scala version is not properly installed, you'll then see a similar exception like below when running the CDM jobs,
Expand Down

0 comments on commit 34e0f11

Please sign in to comment.