-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix path handling for non-conforming datasets #81
Conversation
While rewriting your test I observed that to use a non-compliant data file we have to do something like this: meta = SigMFFile(metadata=ncd_metadata, data_file=data_path)
# tell SigMF that the data is noncompliant
meta.set_global_field(SigMFFile.DATASET_KEY, data_path.name) but really I think the SigMFFile should detect that We can either merge this and create another PR, or if you want you can add this improvement to this PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we never want temp files to exist after testing I used tempfile
and also make some simplifications.
Thanks for all the feedback. I've implemented the automatic detection of NCD discussed above, and also made a small fix in However there's another issue: the behavior of os.remove differs between Windows and Unix platforms (see the linked doc). I'm guessing that you're using Linux where it should work fine, however on Windows (which I'm using) it crashes because the dataset file is still being used by Since I've also noticed another Windows/Linux problem elsewere, I've opened issue #89 regarding this. |
@@ -558,6 +558,12 @@ def set_data_file(self, data_file=None, data_buffer=None, skip_checksum=False, o | |||
self._memmap = raveled.reshape(mapped_reshape) | |||
self.shape = self._memmap.shape if (self._return_type is None) else self._memmap.shape[:-1] | |||
|
|||
if self.data_file is not None: | |||
file_name = path.split(self.data_file)[1] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't love path handling like this, but I opened #90 and we can address it in the future.
Thank you @GreenK173 for your work on NCD handling and this PR. |
This PR replaces PR #79 because I accidentally created that one on the main branch of my fork.
Fixes #78. The matter was also recently dealt with in the SigMF specification repo in issue #321 and PR #322. In particular this PR addresses the issue raised by @lgoix in this comment.
The get_dataset_filename_from_metadata function returns the full file path for non-conforming datasets, not just the file name. Furthermore an unit test verifying this, plus a couple of typos described in the above issue fixed.