spark structured streaming gracefully shutdown

WonYong-Jang · Jan 30, 2024 · 6072979 · 6072979
1 parent 7b8afd7
commit 6072979
Show file tree

Hide file tree

Showing 3 changed files with 2 additions and 7 deletions.
diff --git a/_posts/spark/2021-04-12-Spark-Streaming.md b/_posts/spark/2021-04-12-Spark-Streaming.md
@@ -199,7 +199,7 @@ ssc.awaitTermination()
 즉, 한번 시작하면 명시적인 종료 또는 에러가 없다면 어플리케이션이 임의로 종료되지 않아야 하기 때문이다.   
 
 또한, 종료는 sparkStreamContext.stop() 메서드를 이용하면 된다.    
-참고로 데이터 손실 없는 종료는 [링크](https://wonyong-jang.github.io/spark/2021/06/29/Spark-graceful-shutdown.html)를 
+참고로 데이터 손실 없는 종료는 [링크](https://wonyong-jang.github.io/spark/2021/04/19/Spark-Streaming-Graceful-Shutdown.html)를 
 참고하자.   
 
 

diff --git a/_posts/spark/2021-04-19-Spark-Streaming-Graceful-Shutdown.md b/_posts/spark/2021-04-19-Spark-Streaming-Graceful-Shutdown.md
@@ -140,9 +140,6 @@ object GracefulShutdownExample {
 
         println("ssc is stopped!!!!!!!")  
       }
-
-      Thread.sleep(1000)
-
     }
   }
 

diff --git a/_posts/spark/2022-03-07-Spark-Streaming-To-Structured-Streaming.md b/_posts/spark/2022-03-07-Spark-Streaming-To-Structured-Streaming.md
@@ -313,10 +313,8 @@ def stopStreamQuery(query: StreamingQuery, appId: String, awaitTerminationTimeMs
 
           query.stop() // 종료   
       }
-
-      Thread.sleep(1000)
+      query.awaitTermination(awaitTerminationTimeMs)
    }
-   query.awaitTermination(awaitTerminationTimeMs)
 }
 ```
-Original file line number
+Diff line change
@@ Expand Up / @@ -140,9 +140,6 @@ object GracefulShutdownExample { @@
             println("ssc is stopped!!!!!!!")
           }
-          Thread.sleep(1000)
         }
       }
@@ Expand Down @@