Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Corrupted Could not launch the Visual Layer application on your machine because the files generated by fd.run() seems to be corrupted. #343

Open
KevinColemanInc opened this issue Aug 4, 2024 · 1 comment
Open

[Bug]: Corrupted Could not launch the Visual Layer application on your machine because the files generated by fd.run() seems to be corrupted. #343

KevinColemanInc opened this issue Aug 4, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@KevinColemanInc
Copy link

What happened?

it wont start due to corrupted files.

What did you expect to see?

No response

What version of fastdup were you runnning on?

2.7

What version of Python were you running on?

Python 3.10

Operating System

macos 14.6 m1 max

Reproduction steps

import fastdup

print(fastdup.__version__)

fd = fastdup.create(
    work_dir="/Users/kevin/Desktop/work_dir",
    input_dir="/Users/kevin/Desktop/small.export",
)
fd.run(verbose=True)
fd.explore()
~/Desktop $ rm -rf working_dir
~/Desktop $ python app.py

Relevant log output

python app.py





2.7

fastdup By Visual Layer, Inc. 2024. All rights reserved.

A fastdup dataset object was created!

Input directory is set to "/Users/kevin/Desktop/small.export"
Work directory is set to "work_dir"

The next steps are:
   1. Analyze your dataset with the .run() function of the dataset object
   2. Interactively explore your data on your local machine with the .explore() function of the dataset object

For more information, use help(fastdup) or check our documentation https://docs.visual-layer.com/docs/getting-started-with-fastdup.

Fastdup was already applied, use overwrite=True to re-run
2.7

fastdup By Visual Layer, Inc. 2024. All rights reserved.

A fastdup dataset object was created!

Input directory is set to "/Users/kevin/Desktop/small.export"
Work directory is set to "work_dir"

The next steps are:
   1. Analyze your dataset with the .run() function of the dataset object
   2. Interactively explore your data on your local machine with the .explore() function of the dataset object

For more information, use help(fastdup) or check our documentation https://docs.visual-layer.com/docs/getting-started-with-fastdup.

Fastdup was already applied, use overwrite=True to re-run
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3300, in raw_connection
    return self.pool.connect()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/duckdb_engine/__init__.py", line 274, in connect
    conn = duckdb.connect(*cargs, **cparams)
duckdb.duckdb.IOException: IO Error: Could not set lock on file "/Users/kevin/Desktop/work_dir/vl.duckdb": Conflicting lock is held in /opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/bin/python3.10 (PID 11919) by user kevin. See also https://duckdb.org/docs/connect/concurrency

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/fastdup/fastdup_runner/run_checker.py", line 67, in check_existing_dataset
    datasets = DatasetDB.get_all_datasets()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/fastdup/vldbaccess/dataset.py", line 391, in get_all_datasets
    rows = session.execute(sa.text("""SELECT * FROM datasets ;""")).mappings().all()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
    conn = bind.connect()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3276, in connect
    return self._connection_cls(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2440, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3300, in raw_connection
    return self.pool.connect()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/duckdb_engine/__init__.py", line 274, in connect
    conn = duckdb.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (duckdb.duckdb.IOException) IO Error: Could not set lock on file "/Users/kevin/Desktop/work_dir/vl.duckdb": Conflicting lock is held in /opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/bin/python3.10 (PID 11919) by user kevin. See also https://duckdb.org/docs/connect/concurrency
(Background on this error at: https://sqlalche.me/e/20/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/kevin/Desktop/app.py", line 10, in <module>
    fd.explore()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/fastdup/fastdup_controller.py", line 652, in explore
    do_visual_layer(work_dir=self._work_dir, overwrite=False, run_server=True, verbose=verbose)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/fastdup/fastdup_runner/run.py", line 162, in do_visual_layer
    dataset_id = check_existing_dataset()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/photo_dupe3.10/lib/python3.10/site-packages/fastdup/fastdup_runner/run_checker.py", line 73, in check_existing_dataset
    raise ExplorationError(DB_READ_FAILED_MSG)
fastdup.fastdup_runner.utilities.ExplorationError:
Could not launch the Visual Layer application on your machine because the files generated by fd.run() seems to be corrupted.

Try calling fd.run(overwrite=True), make sure it ends successfully and then call fd.explore() again.

For more information, use help(fastdup) or check our documentation https://docs.visual-layer.com/docs/getting-started-with-fastdup.


### Attach a screenshot [Optional]

_No response_

### Contact Details [Optional]

[email protected]
@KevinColemanInc KevinColemanInc added the bug Something isn't working label Aug 4, 2024
@KevinColemanInc
Copy link
Author

I ended up getting this to work on my m1 by:

  1. running each command in REPL ($ python) instead of ($ python app.py)
  2. ensuring my laptop was connected to the internet
  3. deleting the working director (rm -rf working_dir)
  4. overwrite was optional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KevinColemanInc