docs(python): note multiprocessing incompatibility (#2506)

Fixes #2405
lancedb · Jun 21, 2024 · 735f5b2 · 735f5b2
1 parent cbd0d8e
commit 735f5b2
Show file tree

Hide file tree

Showing 2 changed files with 9 additions and 1 deletion.
diff --git a/docs/integrations/pytorch.rst b/docs/integrations/pytorch.rst
@@ -74,3 +74,8 @@ Available samplers:
 
 - :class:`lance.sampler.ShardedFragmentSampler`
 - :class:`lance.sampler.ShardedBatchSampler`
+
+.. warning::
+    For multiprocessing you should probably not use fork as lance is
+    multi-threaded internally and fork and multi-thread do not work well.
+    Refer to `this discussion <https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555>`_.
diff --git a/docs/integrations/tensorflow.rst b/docs/integrations/tensorflow.rst
@@ -88,4 +88,7 @@ workers.
     for batch in ds:
         print(batch)
 
-
+.. warning::
+    For multiprocessing you should probably not use fork as lance is
+    multi-threaded internally and fork and multi-thread do not work well.
+    Refer to `this discussion <https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555>`_.