Create a set of aggregate statistics over a sequence of datasets. Returns a dataset with a single row and uses the same interface group-by-column-agg.

Example:

  (ds-reduce/aggregate
   {:n-elems (ds-reduce/row-count)
    :price-avg (ds-reduce/mean :price)
    :price-sum (ds-reduce/sum :price)
    :price-med (ds-reduce/prob-median :price)
    :price-iqr (ds-reduce/prob-interquartile-range :price)
    :n-dates (ds-reduce/count-distinct :date :int32)}
   [ds-seq])

Create a reducer that will return a set of values.

Get the set of distinct items given you know the space is no larger than int32 space. The optional finalizer allows you to post-process the data.

Group a sequence of datasets by a column and aggregate down into a new dataset.

• colname - Either a single scalar column name or a vector of column names to group by.

• agg-map - map of result column name to reducer. All values in the agg map must be functions from dataset to hamf (non-parallel) reducers. Note that transducer-compatible rf's - such as kixi.mean, are valid hamf reducers.

• ds-seq - Either a single dataset or sequence of datasets.

See also group-by-column-agg-rf.

Options:

• :map-initial-capacity - initial hashmap capacity. Resizing hash-maps is expensive so we would like to set this to something reasonable. Defaults to 10000.
• :index-filter - A function that given a dataset produces a function from long index to boolean, ideally either nil or a java.util.function.LongPredicate. Only indexes for which the index-filter returns true will be added to the aggregation. For very large datasets, this is a bit faster than using filter before the aggregation.

Example:

user> (require '[tech.v3.dataset :as ds])
nil
user> (require '[tech.v3.dataset.reductions :as ds-reduce])
nil
user> (def ds (ds/->dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"
                            {:key-fn keyword}))

#'user/ds
user> (ds-reduce/group-by-column-agg
       :symbol
       {:price-avg (ds-reduce/mean :price)
        :price-sum (ds-reduce/sum :price)}
       ds)
_unnamed [5 3]:

| :symbol |   :price-avg | :price-sum |
|---------|-------------:|-----------:|
|    MSFT |  24.73674797 |    3042.62 |
|    AAPL |  64.73048780 |    7961.85 |
|     IBM |  91.26121951 |   11225.13 |
|    AMZN |  47.98707317 |    5902.41 |
|    GOOG | 415.87044118 |   28279.19 |

user> (def testds (ds/->dataset {:a ["a" "a" "a" "b" "b" "b" "c" "d" "e"]
                                 :b [22   21  22 44  42  44   77 88 99]}))
#'user/testds
user> (ds-reduce/group-by-column-agg
       [:a :b] {:c (ds-reduce/row-count)}
       testds)
_unnamed [7 3]:

| :a | :b | :c |
|----|---:|---:|
|  e | 99 |  1 |
|  a | 21 |  1 |
|  c | 77 |  1 |
|  d | 88 |  1 |
|  b | 44 |  2 |
|  b | 42 |  1 |
|  a | 22 |  2 |

Produce a transduce-compatible rf that will perform the group-by-column-agg pathway. See documentation for group-by-column-agg.

tech.v3.dataset.reductions-test> (def stocks (ds/->dataset "test/data/stocks.csv" {:key-fn keyword}))
#'tech.v3.dataset.reductions-test/stocks
tech.v3.dataset.reductions-test> (transduce (map identity)
                                            (ds-reduce/group-by-column-agg-rf
                                             :symbol
                                             {:n-elems (ds-reduce/row-count)
                                              :price-avg (ds-reduce/mean :price)
                                              :price-sum (ds-reduce/sum :price)
                                              :symbol (ds-reduce/first-value :symbol)
                                              :n-dates (ds-reduce/count-distinct :date :int32)}
                                             {:index-filter (fn [dataset]
                                                              (let [rdr (dtype/->reader (dataset :price))]
                                                                (hamf/long-predicate
                                                                 idx (> (.readDouble rdr idx) 100.0))))})
                                            [stocks stocks stocks])
_unnamed [4 5]:

| :symbol | :n-elems |   :price-avg | :price-sum | :n-dates |
|---------|---------:|-------------:|-----------:|---------:|
|    AAPL |       93 | 160.19096774 |   14897.76 |       31 |
|     IBM |      120 | 111.03775000 |   13324.53 |       40 |
|    AMZN |       18 | 126.97833333 |    2285.61 |        6 |
|    GOOG |      204 | 415.87044118 |   84837.57 |       68 |

Create a double consumer which will produce a mean of the column.

See docs for tech.v3.dataset.reductions.apache-data-sketch/prob-cdf

• k - defaults to 128. This produces a normalized rank error of about 1.7%

See docs for [[tech.v3.dataset.reductions.apache-data-sketch/prob-interquartile-range

• k - defaults to 128. This produces a normalized rank error of about 1.7%

See docs for tech.v3.dataset.reductions.apache-data-sketch/prob-median

• k - defaults to 128. This produces a normalized rank error of about 1.7%

See docs for tech.v3.dataset.reductions.apache-data-sketch/prob-quantile

• k - defaults to 128. This produces a normalized rank error of about 1.7%

See docs for tech.v3.dataset.reductions.apache-data-sketch/prob-set-cardinality.

Options:

• :hll-lgk - defaults to 12, this is log-base2 of k, so k = 4096. lgK can be from 4 to 21.
• :hll-type - One of #{4,6,8}, defaults to 8. The HLL_4, HLL_6 and HLL_8 represent different levels of compression of the final HLL array where the 4, 6 and 8 refer to the number of bits each bucket of the HLL array is compressed down to. The HLL_4 is the most compressed but generally slightly slower than the other two, especially during union operations.
• :datatype - One of :float64, :int64, :string

Make a group-by-agg reducer.

• column-name - Single column name or multiple columns.
• init-val-fn - Function to produce initial accumulators
• rfn - Function that takes the accumulator and each column's data as as further arguments. For a single-column pathway this looks like a normal clojure reduction function but for two columns it gets extra arguments.
• merge-fn - Function that takes two accumulators and merges them. Merge is not required for group-by-column-agg but it is required for aggregate.
• finalize-fn - finalize the result after aggregation. Optional, will be replaced with identity of not provided.

Given a hamf parallel reducer and a column name, return a dataset reducer of one column.

Calculate a descriptive statistic using reservoir sampling. A list of statistic names are found in tech.v3.datatype.statistics/all-descriptive-stats-names. Options are options used in reservoir-sampler.

Note that this method will not convert datetime objects to milliseconds for you as in descriptive-stats.

Create a simple reducer that returns the number of times reduceIndex was called.

Create a double consumer which will sum the values.