Skip to content

Commit

Permalink
Fixes #443 - this approach preserves metadata on the dataset
Browse files Browse the repository at this point in the history
  • Loading branch information
cnuernber committed Jan 13, 2025
1 parent 7c54599 commit d44f27c
Showing 1 changed file with 19 additions and 3 deletions.
22 changes: 19 additions & 3 deletions src/tech/v3/dataset_api.clj
Original file line number Diff line number Diff line change
Expand Up @@ -216,10 +216,26 @@ user> (ds/rowvec-at stocks -1)
((rowvecs ds) idx))


(defn empty-column-names
"Return a sequence of column names whose empty set length matches the row count of the dataset."
[ds]
(let [rc (row-count ds)]
(->> (columns ds)
(lznc/map #(when (== rc (long (dtype/ecount (missing %))))
(:name (meta %))))
(lznc/remove nil?))))


(defn remove-empty-columns
"Remove all columns that have no data - missing set length equals row count."
[ds]
(remove-columns ds (empty-column-names ds)))


(export-symbols tech.v3.dataset.io
->dataset
->>dataset
write!)
->dataset
->>dataset
write!)


(defn dataset-parser
Expand Down

0 comments on commit d44f27c

Please sign in to comment.