@hackage / hanalyze-frame

Data I/O layer of hanalyze: loaders, cleaning, tidy wrangling

Latest0.2.0.1

About

Metadata

  • Last updated , by frenzieddoll
  • License BSD-3-Clause
  • Categories Mathematics

  • Maintained by: frenzieddoll@gmail.com

  • Lottery factor: 0

Links

Installation

Tested Compilers

  1. 9.6.7

Readme

hanalyze-frame

The data I/O layer of hanalyze. It adopts Hackage's dataframe as the single data representation and covers everything from loading dirty data to tidyverse-style wrangling.

It depends on hanalyze-core plus dataframe-* / cassava / regex-tdfa and friends. Together with -bayes it sits directly on top of core, and the upper layers (-models / -design / -viz) all assume the data representation defined here.

Main modules (14 in total)

Loading (Hanalyze.DataIO.*)
Module Role
DataIO.CSV CSV / TSV / SSV loaders returning a DataFrame directly. loadAuto dispatches on the extension; loadAutoSafe is the defensive variant returning Either plus a log
DataIO.Sniff Guesses delimiter, comment marker, header presence and NA tokens from the first 8 KB
DataIO.Health Flags suspicious patterns in a loaded DataFrame as warning codes W001–W008
DataIO.Clean A per-column cleaning DSL that turns Health warnings into numeric rules
DataIO.Log Structured warnings shared by loaders and preprocessing (LogEntry / LogReport)
DataIO.External Parquet / JSON loaders (via dataframe)
DataIO.Convert Safe extraction of numeric / text columns from a DataFrame into Vector
Wrangling (Hanalyze.Data.* / DataIO.*)
Module Role
Data.Wrangle dplyr-style summarise / mutate / groupBy, DataFrame in and out. Designed symmetrically with hgg's pipe notation
Data.Transform dplyr-style ranking / offsets / cumulatives / binning as pure [a] -> [b]
Data.Factor forcats-style factor type and level operations (fct_*)
Data.Strings stringr-style pure Text operations (str_*)
Data.ColumnSource Minimal "column name → numeric column" abstraction (plot-independent)
DataIO.Reshape Reshape operations dataframe lacks (pivotWider / oneHot / lag & lead / rolling)
DataIO.Preprocess Missing-value detection, removal and imputation / column selection / derived columns / melt

Using it standalone

build-depends: hanalyze-frame, dataframe-core
{-# LANGUAGE OverloadedStrings #-}
import           Hanalyze.DataIO.CSV  (loadAuto)
import           Hanalyze.Data.Wrangle
import           DataFrame.Operators         ((|>))

main = do
  Right df <- loadAuto "flights.csv"      -- IO (Either ParseError DataFrame)
  let out = df |> groupBy ["month"]
               |> summarise [ "mean" =: meanOf "dep_delay"
                            , "q95"  =: quantileOf 0.95 "dep_delay"
                            , "n"    =: nOf ]
  print out
  -- month |        mean        |        q95         |  n
  -- ------|--------------------|--------------------|----
  -- 1     | 4.5                | 11.549999999999999 | 3
  -- 2     | 11.166666666666666 | 23.25              | 3

Aggregators drop NAs by default (dplyr's na.rm = TRUE), and groups come out in ascending key order.

Normally you would just depend on the umbrella package hanalyze and reach these through import Hanalyze. Naming a layer directly is only worth it when you want to minimise dependencies.

← repository README