tech.ml.dataset Walkthrough
tech.ml.dataset (TMD) is a Clojure library designed to ease working with tabular data, similar to data.table in R or Python's Pandas. TMD takes inspiration from the design of those tools, but does not aim to copy their functionality. Instead, TMD is a building block that increases Clojure's already considerable data processing power.
High Level Design
In TMD, a dataset is logically a map of column name to column data. Column data is typed (e.g., a column of 16 bit integers, or a column of 64 bit floating point numbers), similar to a database. Column names may be any Java object - keywords and strings are typical - and column values may be any Java primitive type, or type supported by tech.datatype, datetimes, or arbitrary objects. Column data is stored contiguously in JVM arrays, and missing values are indicated with bitsets.
The outline of the walkthrough follows typical Clojure workflows, showing how those ideas are expanded by TMD:
- Dataset creation from typical formats (csv, tsv, etc...), or sequences of maps, or maps of column name to column values
- REPL-friendly printing of datasets
- Access to dataset (column) values, and working with missing values
- Selecting rows or columns of a dataset, or both at once
- Adding, removing, and updating columns
- Statistical analysis of dataset data
- Functions familiar from
clojure.core(e.g.,sort-by,filter,group-by, etc...) that operate on datasets - Efficient elementwise operations and their use in adding or updating columns derived from data in the dataset
- Getting data back out of datasets for downstream consumers
Dataset Creation
->dataset, ->>dataset
Creating a dataset from .edn (sequence of maps):
user> (require '[tech.v3.dataset :as ds])
nil
user> (ds/->dataset [{:a 1 :b 2} {:a 2 :c 3}])
_unnamed [2 3]:
| :a | :b | :c |
|----|----|----|
| 1 | 2 | |
| 2 | | 3 |
Passing options (as the first argument or the last, whichever is convenient):
user> (ds/->dataset [{:a 1 :b 2} {:a 2 :c 3}] {:dataset-name "simple"})
simple [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
user> (ds/->>dataset {:dataset-name "simple"} [{:a 1 :b 2} {:a 2 :c 3}])
simple [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
Creating a dataset from a file:
user> (ds/->dataset "test/data/stocks.csv")
test/data/stocks.csv [560 3]:
| symbol | date | price |
|--------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
| MSFT | 2000-06-01 | 32.54 |
| MSFT | 2000-07-01 | 28.40 |
| MSFT | 2000-08-01 | 28.40 |
| MSFT | 2000-09-01 | 24.53 |
| MSFT | 2000-10-01 | 28.02 |
| ... | ... | ... |
| AAPL | 2009-05-01 | 135.81 |
| AAPL | 2009-06-01 | 142.43 |
| AAPL | 2009-07-01 | 163.39 |
| AAPL | 2009-08-01 | 168.21 |
| AAPL | 2009-09-01 | 185.35 |
| AAPL | 2009-10-01 | 188.50 |
| AAPL | 2009-11-01 | 199.91 |
| AAPL | 2009-12-01 | 210.73 |
| AAPL | 2010-01-01 | 192.06 |
| AAPL | 2010-02-01 | 204.62 |
| AAPL | 2010-03-01 | 223.02 |
->dataset and ->>dataset interpret a String arguments as a file path (URL), and the format is determined automatically on load. TMD has built-in broad support for typical formats.
For text-based formats such as csv and tsv, zipped (e.g., *.csv.zip) and gzipped (e.g., *.tsv.gz) inputs are uncompressed automatically on load.
InputStreams also work instead of a String, this can be convenient when loading from private cloud storage.
While loading, TMD guesses at column datatypes, and generally does an ok job. If the guesses turn out to be sub-optimal, they can be overridden.
The same functions also load datasets from the internet:
user> (def ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"))
#'user/ds
user> (type ds)
tech.v3.dataset.impl.dataset.Dataset
user> (ds/row-count ds)
1460
Parsing Options
The ->dataset docstring documents the many options available for parsing:
Limiting rows and columns at parse time:
user> (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"
{:column-whitelist ["SalePrice" "1stFlrSF" "2ndFlrSF"]
:n-records 5})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 3]:
| SalePrice | 1stFlrSF | 2ndFlrSF |
|-----------|----------|----------|
| 208500 | 856 | 854 |
| 181500 | 1262 | 0 |
| 223500 | 920 | 866 |
| 140000 | 961 | 756 |
| 250000 | 1145 | 1053 |
Forcing a datatype for a column before parsing:
user> (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"
{:column-whitelist ["SalePrice" "1stFlrSF" "2ndFlrSF"]
:n-records 5
:parser-fn {"SalePrice" :float32}})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [4 3]:
| SalePrice | 1stFlrSF | 2ndFlrSF |
|-----------|----------|----------|
| 208500.0 | 856 | 854 |
| 181500.0 | 1262 | 0 |
| 223500.0 | 920 | 866 |
| 140000.0 | 961 | 756 |
| 250000.0 | 1145 | 1053 |
Loading from .xlsx:
user> (require '[tech.v3.libs.fastexcel]) ;; for xlsx support
nil
user> (def ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"))
#'user/ds
user> ds
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|-----:|------------|-------:|
| 1.0 | Dulce | Abril | Female | United States | 32.0 | 15/10/2017 | 1562.0 |
| 2.0 | Mara | Hashimoto | Female | Great Britain | 25.0 | 16/08/2016 | 1582.0 |
| 3.0 | Philip | Gent | Male | France | 36.0 | 21/05/2015 | 2587.0 |
| 4.0 | Kathleen | Hanner | Female | United States | 25.0 | 15/10/2017 | 3549.0 |
| 5.0 | Nereida | Magwood | Female | United States | 58.0 | 16/08/2016 | 2468.0 |
| 6.0 | Gaston | Brumm | Male | United States | 24.0 | 21/05/2015 | 2554.0 |
| 7.0 | Etta | Hurn | Female | Great Britain | 56.0 | 15/10/2017 | 3598.0 |
| 8.0 | Earlean | Melgar | Female | United States | 27.0 | 16/08/2016 | 2456.0 |
| 9.0 | Vincenza | Weiland | Female | United States | 40.0 | 21/05/2015 | 6548.0 |
| 10.0 | Fallon | Winward | Female | Great Britain | 28.0 | 16/08/2016 | 5486.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990.0 | Chase | Karner | Male | United States | 37.0 | 15/10/2017 | 2154.0 |
| 991.0 | Tommie | Underdahl | Male | United States | 26.0 | 16/08/2016 | 3265.0 |
| 992.0 | Dorcas | Darity | Female | United States | 37.0 | 21/05/2015 | 8765.0 |
| 993.0 | Angel | Sanor | Male | France | 24.0 | 15/10/2017 | 3259.0 |
| 994.0 | Willodean | Harn | Female | United States | 39.0 | 16/08/2016 | 3567.0 |
| 995.0 | Weston | Martina | Male | United States | 26.0 | 21/05/2015 | 6540.0 |
| 996.0 | Roma | Lafollette | Female | United States | 34.0 | 15/10/2017 | 2654.0 |
| 997.0 | Felisa | Cail | Female | United States | 28.0 | 16/08/2016 | 6525.0 |
| 998.0 | Demetria | Abbey | Female | United States | 32.0 | 21/05/2015 | 3265.0 |
| 999.0 | Jeromy | Danz | Male | United States | 39.0 | 15/10/2017 | 3265.0 |
| 1000.0 | Rasheeda | Alkire | Female | United States | 29.0 | 16/08/2016 | 6125.0 |
Notably, here, the "Date" column ended up as a :string due to the funny dd/MM/yyyy format:
user> (meta (get ds "Date"))
{:categorical? true, :name "Date", :datatype :string, :n-elems 1000}
This can be overridden at parse time to get more robust types:
user> (def ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"
{:parser-fn {"Date" [:local-date "dd/MM/yyyy"]}}))
#'user/ds
user> ds
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|-----:|------------|-------:|
| 1.0 | Dulce | Abril | Female | United States | 32.0 | 2017-10-15 | 1562.0 |
| 2.0 | Mara | Hashimoto | Female | Great Britain | 25.0 | 2016-08-16 | 1582.0 |
| 3.0 | Philip | Gent | Male | France | 36.0 | 2015-05-21 | 2587.0 |
| 4.0 | Kathleen | Hanner | Female | United States | 25.0 | 2017-10-15 | 3549.0 |
| 5.0 | Nereida | Magwood | Female | United States | 58.0 | 2016-08-16 | 2468.0 |
| 6.0 | Gaston | Brumm | Male | United States | 24.0 | 2015-05-21 | 2554.0 |
| 7.0 | Etta | Hurn | Female | Great Britain | 56.0 | 2017-10-15 | 3598.0 |
| 8.0 | Earlean | Melgar | Female | United States | 27.0 | 2016-08-16 | 2456.0 |
| 9.0 | Vincenza | Weiland | Female | United States | 40.0 | 2015-05-21 | 6548.0 |
| 10.0 | Fallon | Winward | Female | Great Britain | 28.0 | 2016-08-16 | 5486.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990.0 | Chase | Karner | Male | United States | 37.0 | 2017-10-15 | 2154.0 |
| 991.0 | Tommie | Underdahl | Male | United States | 26.0 | 2016-08-16 | 3265.0 |
| 992.0 | Dorcas | Darity | Female | United States | 37.0 | 2015-05-21 | 8765.0 |
| 993.0 | Angel | Sanor | Male | France | 24.0 | 2017-10-15 | 3259.0 |
| 994.0 | Willodean | Harn | Female | United States | 39.0 | 2016-08-16 | 3567.0 |
| 995.0 | Weston | Martina | Male | United States | 26.0 | 2015-05-21 | 6540.0 |
| 996.0 | Roma | Lafollette | Female | United States | 34.0 | 2017-10-15 | 2654.0 |
| 997.0 | Felisa | Cail | Female | United States | 28.0 | 2016-08-16 | 6525.0 |
| 998.0 | Demetria | Abbey | Female | United States | 32.0 | 2015-05-21 | 3265.0 |
| 999.0 | Jeromy | Danz | Male | United States | 39.0 | 2017-10-15 | 3265.0 |
| 1000.0 | Rasheeda | Alkire | Female | United States | 29.0 | 2016-08-16 | 6125.0 |
user> (meta (get ds "Date"))
{:name "Date", :datatype :local-date, :n-elems 1000}
user> (first (get ds "Date"))
#object[java.time.LocalDate 0x36b45eff "2017-10-15"]
In general, supplying tuples of [datatype parse-fn] allows overriding. As shown above, for datetime types, parse-fn can additionally be a DateTimeFormat format string or a DateTimeFormat object.
A more complete example (necessary, because excel is lax with serialized types):
user> (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"
{:parser-fn {"Date" [:local-date "dd/MM/yyyy"]
"Id" :int32
"column-0" :int32
"Age" :int16}})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|----:|------------|-----:|
| 1 | Dulce | Abril | Female | United States | 32 | 2017-10-15 | 1562 |
| 2 | Mara | Hashimoto | Female | Great Britain | 25 | 2016-08-16 | 1582 |
| 3 | Philip | Gent | Male | France | 36 | 2015-05-21 | 2587 |
| 4 | Kathleen | Hanner | Female | United States | 25 | 2017-10-15 | 3549 |
| 5 | Nereida | Magwood | Female | United States | 58 | 2016-08-16 | 2468 |
| 6 | Gaston | Brumm | Male | United States | 24 | 2015-05-21 | 2554 |
| 7 | Etta | Hurn | Female | Great Britain | 56 | 2017-10-15 | 3598 |
| 8 | Earlean | Melgar | Female | United States | 27 | 2016-08-16 | 2456 |
| 9 | Vincenza | Weiland | Female | United States | 40 | 2015-05-21 | 6548 |
| 10 | Fallon | Winward | Female | Great Britain | 28 | 2016-08-16 | 5486 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990 | Chase | Karner | Male | United States | 37 | 2017-10-15 | 2154 |
| 991 | Tommie | Underdahl | Male | United States | 26 | 2016-08-16 | 3265 |
| 992 | Dorcas | Darity | Female | United States | 37 | 2015-05-21 | 8765 |
| 993 | Angel | Sanor | Male | France | 24 | 2017-10-15 | 3259 |
| 994 | Willodean | Harn | Female | United States | 39 | 2016-08-16 | 3567 |
| 995 | Weston | Martina | Male | United States | 26 | 2015-05-21 | 6540 |
| 996 | Roma | Lafollette | Female | United States | 34 | 2017-10-15 | 2654 |
| 997 | Felisa | Cail | Female | United States | 28 | 2016-08-16 | 6525 |
| 998 | Demetria | Abbey | Female | United States | 32 | 2015-05-21 | 3265 |
| 999 | Jeromy | Danz | Male | United States | 39 | 2017-10-15 | 3265 |
| 1000 | Rasheeda | Alkire | Female | United States | 29 | 2016-08-16 | 6125 |
Map of Columns Format
Creating a dataset from a map of column name to column data:
user> (ds/->dataset {:name ["fred" "ethel" "sally"]
:age [42 42 21]
:likes ["pizza" "sushi" "opera"]})
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
TMD interprets untyped column data (like a persistent vector of objects for example) as either string or double depending on the first entry of the column data sequence. In contrast, typed column data (e.g., a Java array of some primitive type, or a typed tech.v3.datatype container) undergoes no interpretation.
REPL Friendly Printing
In general, it is safe to print out very large datasets or columns. By default, TMD will only print out around twenty values, eliding with ellipses as necessary. This turns out to be super-useful, for example when getting a feel for the data in a particular dataset or column.
By default, datasets with more than 20 rows print abbreviated:
user> (ds/->dataset (for [i (range 21)] {:a (rand)}))
_unnamed [21 1]:
| :a |
|-----------:|
| 0.61372362 |
| 0.17076240 |
| 0.99000549 |
| 0.75449856 |
| 0.95344183 |
| 0.06785647 |
| 0.85751960 |
| 0.29387977 |
| 0.12027818 |
| 0.31456890 |
| ... |
| 0.08843911 |
| 0.56787410 |
| 0.79682279 |
| 0.56327258 |
| 0.32338803 |
| 0.95067985 |
| 0.74832399 |
| 0.10802058 |
| 0.80990261 |
| 0.81393921 |
| 0.42608164 |
The helpful print-all function overrides this so every row is printed:
user> (-> (ds/->dataset (for [i (range 21)] {:a (rand)}))
(ds/print-all))
_unnamed [21 1]:
| :a |
|-----------:|
| 0.42679967 |
| 0.21603676 |
| 0.62454379 |
| 0.20501915 |
| 0.01175086 |
| 0.33983128 |
| 0.25639060 |
| 0.08791288 |
| 0.35557448 |
| 0.65832242 |
| 0.70054068 |
| 0.23177467 |
| 0.84374634 |
| 0.52497340 |
| 0.13117490 |
| 0.87235367 |
| 0.69944234 |
| 0.71521474 |
| 0.13842164 |
| 0.41571045 |
| 0.36734680 |
Note that print-all does not actually print anything, it merely changes the metadata on the dataset to override the default (abbreviated) printing behavior.
Multiline printing:
user> (require '[tech.v3.tensor :as dtt]) ;; Has some types that print nicely on multiple lines
nil
user> (def t (dtt/->tensor (partition 3 (range 9))))
#'user/t
user> (ds/->dataset [{:a 1 :b t}{:a 2 :b t}])
_unnamed [2 2]:
| :a | :b |
|---:|------------------------------|
| 1 | #tech.v3.tensor<object>[3 3] |
| | [[0 1 2] |
| | [3 4 5] |
| | [6 7 8]] |
| 2 | #tech.v3.tensor<object>[3 3] |
| | [[0 1 2] |
| | [3 4 5] |
| | [6 7 8]] |
Overriding this via metadata:
user> (def ds *1)
#'user/ds
user> (with-meta ds
(assoc (meta ds) :print-line-policy :single))
_unnamed [2 2]:
| :a | :b |
|---:|------------------------------|
| 1 | #tech.v3.tensor<object>[3 3] |
| 2 | #tech.v3.tensor<object>[3 3] |
Especially useful when dealing with datasets that may have large amounts of per-column data:
user> (def events-ds (-> (ds/->dataset "https://api.github.com/events"
{:key-fn keyword
:file-type :json})
(vary-meta assoc
:print-line-policy :single
:print-column-max-width 25)))
#'user/events-ds
user> (ds/head events-ds)
https://api.github.com/events [5 8]:
| :id | :type | :actor | :repo | :payload | :public | :created_at | :org |
|-------------|------------------------|----------------|-----------------|-----------------------|---------|----------------------|----------------|
| 13911736787 | PushEvent | {:id 29139614, | {:id 253259114, | {:push_id 5888739305, | true | 2020-10-20T17:49:36Z | |
| 13911736794 | IssuesEvent | {:id 47793873, | {:id 240806054, | {:action "opened", | true | 2020-10-20T17:49:36Z | {:id 61098177, |
| 13911736759 | PushEvent | {:id 71535163, | {:id 304746399, | {:push_id 5888739282, | true | 2020-10-20T17:49:36Z | |
| 13911736795 | PullRequestReviewEvent | {:id 47063667, | {:id 305218173, | {:action "created", | true | 2020-10-20T17:49:36Z | |
| 13911736760 | PushEvent | {:id 22623307, | {:id 287289752, | {:push_id 5888739280, | true | 2020-10-20T17:49:36Z | |
Markdown
Output is printed as markdown tables:
So, this:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
Renders nicely under markdown processing:
| :name | :age | :likes |
|---|---|---|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
The full list of possible printing options is provided in the documentation for dataset-data->str.
Access To Dataset (Column) Values
Datasets are Maps
Every Dataset implements clojure.lang.IPersistentMap. However, unlike Clojure's persistent maps, dataset columns are ordered.
Investigating a dataset as a map, and locating missing values:
user> (def ds (ds/->dataset [{:a 1 :b 2} {:a 2 :c 3}]))
#'user/ds
user> ds
_unnamed [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
user> (first ds)
[:a #tech.v3.dataset.column<int64>[2]
:a
[1, 2]]
user> (second ds)
[:b #tech.v3.dataset.column<int64>[2]
:b
[2, ]]
user> (ds :b)
#tech.v3.dataset.column<int64>[2]
:b
[2, ]
user> (ds/missing (ds :b))
{1}
user> (ds/column-names ds)
(:a :b :c)
user> (for [k (ds/column-names ds)]
[k (ds/missing (ds k))])
([:a {}] [:b {1}] [:c {0}])
Columns are Indexed
Columns implement clojure.lang.Indexed (and so provide nth) and also implement clojure.lang.IFn just like Clojure's persistent vectors.
Getting values is easy:
user> (def lucy-ds (ds/->dataset {:name ["fred" "ethel" "sally"]
:age [42 42 21]
:likes ["pizza" "sushi" "opera"]}))
#'user/lucy-ds
user> (lucy-ds :name)
#tech.v3.dataset.column<string>[3]
:name
[fred, ethel, sally]
user> (nth (lucy-ds :name) 0)
"fred"
user> ((lucy-ds :name) 0)
"fred"
user> ((lucy-ds :name) 2)
"sally"
As Clojure Data
Accessing a dataset's rows as a sequence of maps (with one key per column):
user> (ds/rows lucy-ds)
[{:name "fred", :age 42, :likes "pizza"}
{:name "ethel", :age 42, :likes "sushi"}
{:name "sally", :age 21, :likes "opera"}]
Accessing a dataset's rows as a sequence of vectors (with one entry per column):
user> (ds/rowvecs lucy-ds)
[["fred" 42 "pizza"]
["ethel" 42 "sushi"]
["sally" 21 "opera"]]
These are just views on the dataset, and so accessing these does not double RAM usage, and a Clojureist's intuition about immutability, persistence, and structural sharing are respected (when possible).
Selecting Dataset Rows or Columns - (Subrect Selection)
A dataset with many columns:
user> (def ames-ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"))
#'user/ames-ds
user> (ds/column-count ames-ds)
81
user> (ds/column-names ames-ds)
("Id"
"MSSubClass"
"MSZoning"
...)
user> (get ames-ds "OverallQual")
#tech.v3.dataset.column<int16>[1460]
OverallQual
[7, 6, 7, 7, 8, 5, 8, 7, 7, 5, 5, 9, 5, 7, 6, 7, 6, 4, 5, 5...]
Selecting - and implicitly reordering (!) - columns:
user> (ds/select-columns ames-ds ["Id" "OverallQual" "SalePrice"])
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| Id | OverallQual | SalePrice |
|-----:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 3 | 7 | 223500 |
| 4 | 7 | 140000 |
| 5 | 8 | 250000 |
| 6 | 5 | 143000 |
| 7 | 8 | 307000 |
| 8 | 7 | 200000 |
| 9 | 7 | 129900 |
| 10 | 5 | 118000 |
| ... | ... | ... |
| 1450 | 5 | 92000 |
| 1451 | 5 | 136000 |
| 1452 | 8 | 287090 |
| 1453 | 5 | 145000 |
| 1454 | 5 | 84500 |
| 1455 | 7 | 185000 |
| 1456 | 6 | 175000 |
| 1457 | 6 | 210000 |
| 1458 | 7 | 266500 |
| 1459 | 5 | 142125 |
| 1460 | 5 | 147500 |
Interest in just a few specific rows:
user> (-> (ds/select-columns ames-ds ["Id" "OverallQual" "SalePrice"])
(ds/select-rows [0 1 3 5 17 42]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 3]:
| Id | OverallQual | SalePrice |
|---:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 4 | 7 | 140000 |
| 6 | 5 | 143000 |
| 18 | 4 | 90000 |
| 43 | 5 | 144000 |
Or the same as a one-liner:
user> (ds/select ames-ds ["Id" "OverallQual" "SalePrice"] [0 1 3 5 17 42])
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 3]:
| Id | OverallQual | SalePrice |
|---:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 4 | 7 | 140000 |
| 6 | 5 | 143000 |
| 18 | 4 | 90000 |
| 43 | 5 | 144000 |
Adding, Removing, and Updating Columns
Because datasets are maps (of column name to column data), adding a sequence as a column is trivial:
user> (assoc lucy-ds :foo (repeatedly #(rand)))
_unnamed [3 4]:
| :name | :age | :likes | :foo |
|-------|-----:|--------|-----------:|
| fred | 42 | pizza | 0.21302376 |
| ethel | 42 | sushi | 0.40027647 |
| sally | 21 | opera | 0.04117065 |
Similarly, removing columns is just dissoc:
user> (dissoc lucy-ds :age)
_unnamed [3 2]:
| :name | :likes |
|-------|--------|
| fred | pizza |
| ethel | sushi |
| sally | opera |
Look at these last two examples again, they subtly indicate that TMD datasets are functional... Neither operation changed lucy-ds.
🕉
The powerful row-map function is often the best way to add (or update) columns derived from data already in the dataset:
user> (-> (ds/select ames-ds ["Id" "OverallQual" "SalePrice"] [0 1 3 5 17 42])
(ds/row-map (fn [{:strs [OverallQual SalePrice]}]
{"PricePerQual" (quot SalePrice OverallQual)
"BetterPrice" (* 2 SalePrice)})))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 5]:
| Id | OverallQual | SalePrice | PricePerQual | BetterPrice |
|---:|------------:|----------:|-------------:|------------:|
| 1 | 7 | 208500 | 29785 | 417000 |
| 2 | 6 | 181500 | 30250 | 363000 |
| 4 | 7 | 140000 | 20000 | 280000 |
| 6 | 5 | 143000 | 28600 | 286000 |
| 18 | 4 | 90000 | 22500 | 180000 |
| 43 | 5 | 144000 | 28800 | 288000 |
Another example of adding a column:
user> (require '[tech.v3.datatype.functional :as dfn]) ;; for functional column operation
nil
user> lucy-ds
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
user> (dfn/log (:age lucy-ds))
[3.7376696182833684 3.7376696182833684 3.044522437723423]
user> (assoc lucy-ds :log-age (dfn/log (:age lucy-ds)))
_unnamed [3 4]:
| :name | :age | :likes | :log-age |
|-------|-----:|--------|-----------:|
| fred | 42 | pizza | 3.73766962 |
| ethel | 42 | sushi | 3.73766962 |
| sally | 21 | opera | 3.04452244 |
Constants are broadcast to each row:
user> (assoc lucy-ds :spring-chicken false)
_unnamed [3 4]:
| :name | :age | :likes | :spring-chicken |
|-------|-----:|--------|-----------------|
| fred | 42 | pizza | false |
| ethel | 42 | sushi | false |
| sally | 21 | opera | false |
Statistical Analysis
The tech.v3.datatype.statistics namespace from the underlying dtype-next system provides many useful statistical functions:
user> (require '[tech.v3.datatype.statistics :as stats])
nil
user> (def stock-ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"
{:key-fn keyword}))
#'user/stock-ds
user> (ds/filter-column stock-ds :symbol #{"MSFT"})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [123 3]:
| :symbol | :date | :price |
|---------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
| MSFT | 2000-06-01 | 32.54 |
| MSFT | 2000-07-01 | 28.40 |
| MSFT | 2000-08-01 | 28.40 |
| MSFT | 2000-09-01 | 24.53 |
| MSFT | 2000-10-01 | 28.02 |
| ... | ... | ... |
| MSFT | 2009-05-01 | 20.59 |
| MSFT | 2009-06-01 | 23.42 |
| MSFT | 2009-07-01 | 23.18 |
| MSFT | 2009-08-01 | 24.43 |
| MSFT | 2009-09-01 | 25.49 |
| MSFT | 2009-10-01 | 27.48 |
| MSFT | 2009-11-01 | 29.27 |
| MSFT | 2009-12-01 | 30.34 |
| MSFT | 2010-01-01 | 28.05 |
| MSFT | 2010-02-01 | 28.67 |
| MSFT | 2010-03-01 | 28.80 |
user> (def prices (:price (ds/filter-column stock-ds :symbol #{"MSFT"})))
#'user/prices
user> (stats/min prices)
15.81
user> (stats/mean prices)
24.736747967479673
user> (stats/max prices)
43.22
The descriptive-stats function calculates many statistics at once, returning the result as a dataset:
user> (ds/descriptive-stats (ds/filter-column stock-ds :symbol #{"MSFT"}))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| :symbol | :string | 123 | 0 | | | MSFT | | | | MSFT | MSFT |
| :date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| :price | :float64 | 123 | 0 | 15.81 | 24.74 | | 43.22 | 4.30395786E+00 | 1.16559225 | 39.81 | 28.80 |
The same data is available as a sequence of maps (with one map per column) from brief:
user> (ds/brief (ds/filter -column stock-ds :symbol #{"MSFT"}))
({:n-missing 0,
:col-name :symbol,
:histogram (["MSFT" 123]),
:datatype :string,
:mode "MSFT",
:n-valid 123,
:values ["MSFT"],
:first "MSFT",
:last "MSFT",
:n-values 1}
{:min #object[java.time.LocalDate 0x3059f98a "2000-01-01"],
:n-missing 0,
:col-name :date,
:mean #object[java.time.LocalDate 0x78193b42 "2005-01-30"],
:datatype :packed-local-date,
:skew 2.5335340939019964E-4,
:standard-deviation 9.375545376336331E10,
:quartile-3 #object[java.time.LocalDate 0x2a73389f "2007-09-01"],
:n-valid 123,
:quartile-1 #object[java.time.LocalDate 0x5818d8e "2002-07-01"],
:median #object[java.time.LocalDate 0x3eaaea8d "2005-02-01"],
:max #object[java.time.LocalDate 0x4aeeb094 "2010-03-01"],
:first #object[java.time.LocalDate 0x16291c62 "2000-01-01"],
:last #object[java.time.LocalDate 0x56533361 "2010-03-01"]}
{:min 15.81,
:n-missing 0,
:col-name :price,
:mean 24.736747967479673,
:datatype :float64,
:skew 1.165592245374292,
:standard-deviation 4.303957861320731,
:quartile-3 27.34,
:n-valid 123,
:quartile-1 21.75,
:median 24.11,
:max 43.22,
:first 39.81,
:last 28.8})
Supremely useful for developing generalized systems that work on arbitrary datasets.
Sort-by, Filter, Group-by
While conceptually similar to the ones in clojure.core, these functions are aware of the tabular/columnar dataset structure, and so can be much more efficient (especially when using the single-column versions).
When using the whole-dataset versions, TMD passes each row (as a map) into the argument function, this matches the expectation of Clojure programmers:
user> (-> ames-ds
(ds/filter #(< 30000 (get % "SalePrice")))
(ds/select ["OverallQual" "SalePrice"] (range 5)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| OverallQual | SalePrice |
|------------:|----------:|
| 7 | 208500 |
| 6 | 181500 |
| 7 | 223500 |
| 7 | 140000 |
| 8 | 250000 |
Operating on a single column works the same, and is faster, so do that when possible:
user> (-> ames-ds
(ds/filter-column "SalePrice" #(< 30000 %))
(ds/select ["OverallQual" "SalePrice"] (range 5)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| OverallQual | SalePrice |
|------------:|----------:|
| 7 | 208500 |
| 6 | 181500 |
| 7 | 223500 |
| 7 | 140000 |
| 8 | 250000 |
Sorting
Sorting (again, like in Clojure):
user> (-> (ds/sort-by-column ames-ds "SalePrice" >)
(ds/select-columns ["Id" "OverallQual" "SalePrice"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| Id | OverallQual | SalePrice |
|-----:|------------:|----------:|
| 692 | 10 | 755000 |
| 1183 | 10 | 745000 |
| 1170 | 10 | 625000 |
| 899 | 9 | 611657 |
| 804 | 9 | 582933 |
| 1047 | 9 | 556581 |
| 441 | 10 | 555000 |
| 770 | 8 | 538000 |
| 179 | 9 | 501837 |
| 799 | 9 | 485000 |
| ... | ... | ... |
| 1381 | 3 | 58500 |
| 813 | 5 | 55993 |
| 706 | 4 | 55000 |
| 1326 | 4 | 55000 |
| 1338 | 4 | 52500 |
| 711 | 3 | 52000 |
| 31 | 4 | 40000 |
| 534 | 1 | 39300 |
| 969 | 3 | 37900 |
| 917 | 2 | 35311 |
| 496 | 4 | 34900 |
Grouping
Grouping produces a map of column values to datasets with only rows with that column value:
user> lucy-ds
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
user> (ds/group-by-column lucy-ds :age)
{42 _unnamed [2 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
, 21 _unnamed [1 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| sally | 21 | opera |
}
This structure (a map of data values to data with that value) will be familiar to even the most casual clojure.core/group-by enjoyer.
Descriptive Stats and Group-By and DateTime Types
Automatically parsing dates in a CSV dataset:
user> (def stock-ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"))
#'user/stock-ds
user> (ds/head stock-ds)
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [5 3]:
| symbol | date | price |
|--------|------------|------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
user> (get stock-ds "date")
#tech.v3.dataset.column<packed-local-date>[560]
date
[2000-01-01, 2000-02-01, 2000-03-01, 2000-04-01, 2000-05-01, 2000-06-01, 2000-07-01, 2000-08-01, 2000-09-01, 2000-10-01, 2000-11-01, 2000-12-01, 2001-01-01, 2001-02-01, 2001-03-01, 2001-04-01, 2001-05-01, 2001-06-01, 2001-07-01, 2001-08-01...]
user> (meta (get stock-ds "date"))
{:name "date", :datatype :packed-local-date, :n-elems 560}
Those dates are not strings (!)
So, math on them works as expected:
user> (->> (ds/group-by-column stock-ds "symbol")
(map (fn [[k v]] (ds/descriptive-stats v))))
(https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | MSFT | | | | MSFT | MSFT |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 15.81 | 24.74 | | 43.22 | 4.30395786E+00 | 1.16559225 | 39.81 | 28.80 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | AMZN | | | | AMZN | AMZN |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 5.970 | 47.99 | | 135.9 | 2.88913206E+01 | 0.98217538 | 64.56 | 128.8 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | IBM | | | | IBM | IBM |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 53.01 | 91.26 | | 130.3 | 1.65133647E+01 | 0.44446266 | 100.5 | 125.6 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|------------:|------------|------------|
| symbol | :string | 68 | 0 | | | GOOG | | | | GOOG | GOOG |
| date | :packed-local-date | 68 | 0 | 2004-08-01 | 2007-05-17 | | 2010-03-01 | 5.20003989E+10 | 0.00094625 | 2004-08-01 | 2010-03-01 |
| price | :float64 | 68 | 0 | 102.4 | 415.9 | | 707.0 | 1.35069851E+02 | -0.22776524 | 102.4 | 560.2 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | AAPL | | | | AAPL | AAPL |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 7.070 | 64.73 | | 223.0 | 6.31237823E+01 | 0.93215285 | 25.94 | 223.0 |
)
Note:
group-by-columnreturned a map of stock symbols to datasets of stocks with that symbol- On the next line, the function argument to
mapdestructured that as tuples with eachva dataset - Finally
descriptive-statsoperates on those individually, notably doing useful min/mean/max calculations on the dates
Elementwise Operations
The underlying dtype-next system includes a mathematical abstraction designed to work with things like columns.
Using this to create new columns as lazily evaluated combinations of other columns:
user> (def updated-ames
(assoc ames-ds
"TotalBath"
(dfn/+ (ames-ds "BsmtFullBath")
(dfn/* 0.5 (ames-ds "BsmtHalfBath"))
(ames-ds "FullBath")
(dfn/* 0.5 (ames-ds "HalfBath")))))
#'user/updated-ames
user> (ds/head (ds/select-columns updated-ames ["BsmtFullBath" "BsmtHalfBath" "FullBath" "HalfBath" "TotalBath"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 5]:
| BsmtFullBath | BsmtHalfBath | FullBath | HalfBath | TotalBath |
|-------------:|-------------:|---------:|---------:|----------:|
| 1 | 0 | 2 | 1 | 3.5 |
| 0 | 1 | 2 | 0 | 2.5 |
| 1 | 0 | 2 | 1 | 3.5 |
| 1 | 0 | 1 | 0 | 2.0 |
| 1 | 0 | 2 | 1 | 3.5 |
dtype-next's emap enables expressing arbitrary conversions from columns into columns:
user> (require '[tech.v3.datatype :as dtype])
nil
user> (def named-baths
(assoc updated-ames "NamedBath" (dtype/emap #(let [tbaths (double %)]
(cond
(< tbaths 1.0) "almost none"
(< tbaths 2.0) "somewhat doable"
(< tbaths 3.0) "getting somewhere"
:else "living in style"))
:string
(updated-ames "TotalBath"))))
#'user/named-baths
user> (ds/head (ds/select-columns named-baths ["TotalBath" "NamedBath"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| TotalBath | NamedBath |
|----------:|-------------------|
| 3.5 | living in style |
| 2.5 | getting somewhere |
| 3.5 | living in style |
| 2.0 | getting somewhere |
| 3.5 | living in style |
Expensive houses have more bathrooms:
user> (-> (ds/select-columns named-baths ["TotalBath" "NamedBath" "SalePrice"])
(ds/sort-by-column "SalePrice" >))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| TotalBath | NamedBath | SalePrice |
|----------:|-------------------|----------:|
| 4.0 | living in style | 755000 |
| 4.5 | living in style | 745000 |
| 4.5 | living in style | 625000 |
| 3.5 | living in style | 611657 |
| 3.5 | living in style | 582933 |
| 3.5 | living in style | 556581 |
| 3.0 | living in style | 555000 |
| 4.5 | living in style | 538000 |
| 3.0 | living in style | 501837 |
| 3.5 | living in style | 485000 |
| ... | ... | ... |
| 2.0 | getting somewhere | 58500 |
| 1.0 | somewhat doable | 55993 |
| 2.0 | getting somewhere | 55000 |
| 1.0 | somewhat doable | 55000 |
| 1.0 | somewhat doable | 52500 |
| 1.0 | somewhat doable | 52000 |
| 1.0 | somewhat doable | 40000 |
| 1.0 | somewhat doable | 39300 |
| 1.0 | somewhat doable | 37900 |
| 1.0 | somewhat doable | 35311 |
| 1.0 | somewhat doable | 34900 |
DateTime Types
The underlying dtype-next system supports sophisticated datetime processing.
A more real example of summarizing stock prices by year:
user> (def stock-ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"
{:key-fn keyword}))
#'user/stock-ds
user> (ds/head stock-ds)
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [5 3]:
| :symbol | :date | :price |
|---------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
user> (meta (stock-ds :date))
{:name :date, :datatype :packed-local-date, :n-elems 560}
user> (require '[tech.v3.datatype.datetime :as dtype-dt]) ;; for datetime processing
nil
user> (require '[tech.v3.datatype.functional :as dfn]) ;; for functional column processing
nil
user> (assoc stock-ds :year (dtype-dt/long-temporal-field :years (stock-ds :date)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [560 4]:
| :symbol | :date | :price | :year |
|---------|------------|-------:|------:|
| MSFT | 2000-01-01 | 39.81 | 2000 |
| MSFT | 2000-02-01 | 36.35 | 2000 |
| MSFT | 2000-03-01 | 43.22 | 2000 |
| MSFT | 2000-04-01 | 28.37 | 2000 |
| MSFT | 2000-05-01 | 25.45 | 2000 |
| MSFT | 2000-06-01 | 32.54 | 2000 |
| MSFT | 2000-07-01 | 28.40 | 2000 |
| MSFT | 2000-08-01 | 28.40 | 2000 |
| MSFT | 2000-09-01 | 24.53 | 2000 |
| MSFT | 2000-10-01 | 28.02 | 2000 |
| ... | ... | ... | ... |
| AAPL | 2009-05-01 | 135.81 | 2009 |
| AAPL | 2009-06-01 | 142.43 | 2009 |
| AAPL | 2009-07-01 | 163.39 | 2009 |
| AAPL | 2009-08-01 | 168.21 | 2009 |
| AAPL | 2009-09-01 | 185.35 | 2009 |
| AAPL | 2009-10-01 | 188.50 | 2009 |
| AAPL | 2009-11-01 | 199.91 | 2009 |
| AAPL | 2009-12-01 | 210.73 | 2009 |
| AAPL | 2010-01-01 | 192.06 | 2010 |
| AAPL | 2010-02-01 | 204.62 | 2010 |
| AAPL | 2010-03-01 | 223.02 | 2010 |
user> (->> (-> (assoc stock-ds :year (dtype-dt/long-temporal-field :years (stock-ds :date)))
(ds/group-by (juxt :symbol :year)))
;; Now operating on the map of pairs to ds returned by `group-by`
(map (fn [[[symbol year] ds]]
{:symbol symbol
:year year
:mean-price (dfn/mean (ds :price))}))
(sort-by (juxt :symbol :year))
(ds/->>dataset {:dataset-name "Mean Stock Price by Year"})
(ds/print-all))
Mean Stock Price by Year [51 3]:
| :symbol | :year | :mean-price |
|---------|------:|-------------:|
| AAPL | 2000 | 21.74833333 |
| AAPL | 2001 | 10.17583333 |
| AAPL | 2002 | 9.40833333 |
| AAPL | 2003 | 9.34750000 |
| AAPL | 2004 | 18.72333333 |
| AAPL | 2005 | 48.17166667 |
| AAPL | 2006 | 72.04333333 |
| AAPL | 2007 | 133.35333333 |
| AAPL | 2008 | 138.48083333 |
| AAPL | 2009 | 150.39333333 |
| AAPL | 2010 | 206.56666667 |
| AMZN | 2000 | 43.93083333 |
| AMZN | 2001 | 11.73916667 |
| AMZN | 2002 | 16.72333333 |
| AMZN | 2003 | 39.01666667 |
| AMZN | 2004 | 43.26750000 |
| AMZN | 2005 | 40.18750000 |
| AMZN | 2006 | 36.25166667 |
| AMZN | 2007 | 69.95250000 |
| AMZN | 2008 | 69.01500000 |
| AMZN | 2009 | 90.73083333 |
| AMZN | 2010 | 124.21000000 |
| GOOG | 2004 | 159.47600000 |
| GOOG | 2005 | 286.47250000 |
| GOOG | 2006 | 415.25666667 |
| GOOG | 2007 | 548.75833333 |
| GOOG | 2008 | 454.99916667 |
| GOOG | 2009 | 449.92000000 |
| GOOG | 2010 | 538.97666667 |
| IBM | 2000 | 96.91416667 |
| IBM | 2001 | 96.96833333 |
| IBM | 2002 | 75.12500000 |
| IBM | 2003 | 77.30750000 |
| IBM | 2004 | 83.88583333 |
| IBM | 2005 | 77.49750000 |
| IBM | 2006 | 78.71750000 |
| IBM | 2007 | 101.27666667 |
| IBM | 2008 | 107.22500000 |
| IBM | 2009 | 109.29666667 |
| IBM | 2010 | 124.85333333 |
| MSFT | 2000 | 29.67333333 |
| MSFT | 2001 | 25.34750000 |
| MSFT | 2002 | 21.82666667 |
| MSFT | 2003 | 20.93416667 |
| MSFT | 2004 | 22.67416667 |
| MSFT | 2005 | 23.84583333 |
| MSFT | 2006 | 24.75833333 |
| MSFT | 2007 | 29.28416667 |
| MSFT | 2008 | 25.20833333 |
| MSFT | 2009 | 22.87250000 |
| MSFT | 2010 | 28.50666667 |
Writing A Dataset Out
The write! function puts a dataset in a file or on a stream:
(ds/write! test-ds "test.csv")
(ds/write! test-ds out-stream)
(ds/write! test-ds "test.nippy")
The file format is derived from the extension.
The excellent nippy format is a good choice if downstream consumers are other TMD users, in which case datasets work just like any other edn data. While CSV stores numbers as ascii text, nippy stores them binary, saving some space, but the real win is 10X-100X loading performance improvement when compared to csv/tsv.
Moreover, because nippy knows Clojure, you can write out heterogeneous data structures that contain datasets and other things such as the result of a group-by:
user> (require '[taoensso.nippy :as nippy])
nil
user> (def byte-data (nippy/freeze (ds/group-by stock-ds :symbol)))
#'user/byte-data
user> (type byte-data)
[B
user> (keys (nippy/thaw byte-data))
("MSFT" "GOOG" "AAPL" "IBM" "AMZN")
user> (first (nippy/thaw byte-data))
["MSFT"
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [123 3]:
| :symbol | :date | :price |
|---------|------------|--------|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
For Apache arrow support, see the tech.v3.libs.arrow documentation.
For Parquet, see the tech.v3.libs.parquet documentation.
Further Reading
-
The quick reference summarizes many of the most frequently used functions with hints about their use.
-
The API docs list every function available in TMD.