File Coverage

blib/lib/App/RecordStream.pm

Criterion	Covered	Total	%
statement	11	11	100.0
branch			n/a
condition			n/a
subroutine	4	4	100.0
pod	0	1	0.0
total	15	16	93.7

use strict;
use warnings;

package App::RecordStream;
use Module::Pluggable::Object;

our $VERSION = "4.0.23";

sub operation_packages {
  sort { $a cmp $b }
  Module::Pluggable::Object->new(
    search_path => "App::RecordStream::Operation",
    max_depth   => 4,
  )->plugins
}

=encoding utf-8

=for markdown


=for markdown
[![CPAN version](https://badge.fury.io/pl/App-RecordStream.png)](https://metacpan.org/release/App-RecordStream)
[![Build Status](https://travis-ci.org/benbernard/RecordStream.svg?branch=master)](https://travis-ci.org/benbernard/RecordStream)

=head1 NAME

App::RecordStream - recs - A system for command-line analysis of data

=head1 SYNOPSIS

A set of programs for creating, manipulating, and outputting a stream of
Records, or JSON hashes.  Inspired by Monad.

=head1 INSTALLATION

=head2 Quick, standalone bundle

The quickest way to start using recs is via the minimal, standalone bundle:

  curl -fsSL https://recs.pl > recs
  chmod +x recs
  ./recs --help

This is also known as the "fatpacked" recs.

=head2 From CPAN

You can also install recs from L as App::RecordStream:

  cpanm --interactive App::RecordStream

Using L in interactive mode will prompt you for optional feature
support.  Other CPAN clients such as L and L also work fine, but
you can't opt to use any optional features (just like cpanm in non-interactive
mode).  A kitchen-sink install of App::RecordStream looks like:

  cpanm --with-recommends --with-all-features App::RecordStream

If you don't have L itself, you can install it easily with:

  curl -fsSL https://cpanmin.us | perl - App::cpanminus

=head1 DESCRIPTION

The recs system consists of three basic sets of commands:

=over 4

=item * I commands responsible for generating streams of record objects

=item * I commands responsible for analyzing, selecting, and manipulating records

=item * I commands responsible for taking record streams and producing output for humans or other programs

=back

These commands can interface with other systems to retrieve data, parse existing
files, or just regex out some values from a text stream.

Commands are run using C.  If you're using
a CPAN-based install, you may also run commands directly as C,
though this is no longer recommended for forwards compatibility.  Both
L provide a top-level C executable
which dispatches to commands, so this is the preferred invocation style.

The core recs commands are briefly summarized below, and you can list all
available commands by running C.

To read more about each command, run C.  Longer
documentation is available as C or C.
For example, to read more about L, you might run any of the
following:

  recs fromcsv --help
  recs fromcsv --help-all
  perldoc recs-fromcsv

=head1 COMMANDS

=head2 Input Generation

=over 4

=item fromcsv

Produces records from a csv file/stream

=item fromdb

Produces records for a db table, or from a SELECT statement into a db.

=item fromre

Matches input streams against a regex, puts capture groups into hashes

=item frommongo

Generate a record stream from a MongoDB query.

=item frommultire

Matches input streams against several regexes, puts capture groups into the record

=item fromsplit

Splits input stream on a delimiter

=item fromps

Generate records from the process tree

=item fromatomfeed

Produces records for an optionally paginated atom feed.

=item fromxml

Produces records for an XML document.

=item fromkv

Produces records from input streams containing loosely formed key/value pairs

=item fromtcpdump

Produces records from packet capture files (.pcap) as made by tcpdump

=back

=head2 Stream Manipulation

=over 4

=item annotate

Annotate records with common fields, will memoize additions to speed up common
annotations

=item collate

Perforce aggregation operations on records.  Group by a field, get an average,
sum, correlation, etc.  Very powerful

=item delta

Transform values into deltas between adjacent records

=item eval

Eval a string of Perl against each record

=item flatten

Flatten records of input to one level

=item grep

Select records for which a string of Perl evaluates to true.

=item multiplex

Take records, grouped by keys, and run a separate recs command for each group.

=item normalizetime

Based on a time field, tag records with a normalized time, i.e. every 5 minute buckets

=item join

Perform an inner join of two record streams.  Associate records in one stream
with another stream.

=item substream

Filter to a range of matching records with paired Perl snippets C<--start> and C<--end>.

=item sort

Sort records based on keys, may specify multiple levels of sorting, as well as
numerical or lexical sort ordering

=item topn

Outputs the top I records. You may segment the input based on a list of keys
such that unique values of keys are treated as distinct input streams. This
enables top I listings per value groupings.

=item xform

Perform a block of Perl on each record, which may modify the record, Record is
then output

=item generate

Perform a block of Perl on each record to generate a record stream, which is
then output with a chain link back to the original record.

=back

=head2 Output Generation

=over 4

=item todb

Inserts records into a DBI supported SQL database.  Will create a local SQLite
database by default

=item tocsv

Generates correctly quoted CSV files from record streams.

=item tognuplot

Create a graph of field values in a record using GNU Plot.

=item totable

Pretty prints a table of results.

=item tohtml

Prints out an HTML table of the record stream

=item toprettyprint

Prettily prints records, one key to a line, great for making sense of very large records

=item toptable

Prints a multi-dimensional (pivot) table of values.  Very powerful.

=back

=head1 KEY SPECS

Many of the commands above take key arguments to specify or assign to a key in a
record. Almost all of the places where you can specify a key (which normally
means a first level key in the record), you can instead specify a key spec.

A key spec may be nested, and may index into arrays.  Use a C to nest into a
hash and a C<#NUM> to index into an array (i.e. C<#2>)

An example is in order, take a record like this:

  {"biz":["a","b","c"],"foo":{"bar 1":1},"zap":"blah1"}
  {"biz":["a","b","c"],"foo":{"bar 1":2},"zap":"blah2"}
  {"biz":["a","b","c"],"foo":{"bar 1":3},"zap":"blah3"}

In this case a key spec of C would have the values 1, 2, and 3
respectively.

Similarly, C would have the value of C for all 3 records

=head2 Fuzzy matching

You can also prefix key specs with C<@> to engage the fuzzy matching logic.
Matching is tried like this, in order, with the first key to match winning:

=over 4

=item 1. Exact match (C)

=item 2. Prefix match (C)

=item 3. Match anywhere in the key (C)

=back

Given the above example data and the fuzzy key spec C<@b/#2>, the C portion
would expand to C and C<2> would be the index into the array, so all
records would have the value of C.

Simiarly, C<@f/b> would have values 1, 2, and 3.

=head1 WRITING YOUR OWN COMMANDS

The data stream format of the recs commands is JSON hashes separated by new
lines.  If you wish to write your own recs command in your own language, just
get a JSON parser and you should be good to go.  The recs commands use
L.

If you name your command as C and put it somewhere in your
C environment variable, the C command will dispatch to it when
called as C.  It will also be included in C
output.

If you want to write your new command in Perl, you can use the same Perl API
that the standard recs toolkit uses.  See the various
L subclasses.  Once your new operation class is
installed in perl's library paths, C will find it automatically without
the need for any executable command shim.

=head1 EXAMPLES

  # look in the custom access log for all accesses with greater than 5 seconds,
  # display in a table
  cat access.log \
    | recs fromre --fields ip,time '^(\d+).*TIME: (\d+)' \
    | recs grep '$r->{time} > 5' \
    | recs totable

=head1 SEE ALSO

Each of the commands discussed have a C<--help> mode available to print out
usage and examples for the particular command.  See that documentation for
detailed information on the operation of each of the commands.  Also see some
other man pages:

=over

=item * Run C or see L for a set of simple recs examples

=item * Run C or see L for a humorous introduction to RecordStream

=back

=head1 AUTHORS

Benjamin Bernard

Keith Amling

Thomas Sibley

=head1 COPYRIGHT AND LICENSE

Copyright 2007–2017 by the AUTHORS

This software is released under the MIT and Artistic 1.0 licenses.

=cut

1;