feat: Add cum_count_*() operations to update_by and Numeric library

[lbooker42]

Adds new operation to update_by to output the cumulative count of values encountered.

• cum_count_all - cumulative count of all values (including null)
• cum_count_non_null - cumulative count of non-null values
• cum_count_null - cumulative count of null values
• cum_count_neg - cumulative count of non-null negative values
• cum_count_pos - cumulative count of non-null positive values
• cum_count_zero - cumulative count of zero values
• cum_count_nan - cumulative count of NaN values
• cum_count_inf - cumulative count of +Inf/-Inf values
• cum_count_finite - cumulative count of non-null finite values

Example Code

Groovy

t = emptyTable(100).update("A=ii%11==0?null:ii%11-5", "B=(double)A", "C=A==0?Double.NaN:A==2?Double.POSITIVE_INFINITY:A==4?Double.NEGATIVE_INFINITY:(double)A")

opArr = [
    CumCountAll("countA=A","countB=B","countC=C"),
    CumCountNonNull("countA=A","countB=B","countC=C"),
    CumCountNull("countNullA=A","countNullB=B","countNullC=C"),
    CumCountNegative("countNegA=A","countNegB=B","countNegC=C"),
    CumCountPositive("countPosA=A","countPosB=B","countPosC=C"),
    CumCountZero("countZeroA=A","countZeroB=B","countZeroC=C"),
    CumCountNaN("countNaNA=A","countNaNB=B","countNaNC=C"),
    CumCountInfinite("countInfA=A","countInfB=B","countInfC=C"),
    CumCountFinite("countFiniteA=A","countFiniteB=B","countFiniteC=C"),
]

t_count = t.updateBy(opArr)

Python

from deephaven import empty_table
from deephaven.updateby import cum_count, cum_count_null, cum_count_neg, cum_count_pos, cum_count_zero, cum_count_nan, cum_count_inf, cum_count_finite

t = empty_table(100).update(["A=ii%11==0?null:ii%11-5", "B=(double)A","C=A==0?Double.NaN:A==2?Double.POSITIVE_INFINITY:A==4?Double.NEGATIVE_INFINITY:(double)A"])

op_arr = [
    cum_count_all(cols=["count_A=A","count_B=B", "count_C=C"]),
    cum_count_non_null(cols=["count_A=A","count_B=B", "count_C=C"]),
    cum_count_null(cols=["count_null_A=A","count_null_B=B", "count_null_C=C"]),
    cum_count_neg(cols=["count_neg_A=A","count_neg_B=B", "count_neg_C=C"]),
    cum_count_pos(cols=["count_pos_A=A","count_pos_B=B", "count_pos_C=C"]),
    cum_count_zero(cols=["count_zero_A=A","count_zero_B=B", "count_zero_C=C"]),
    cum_count_nan(cols=["count_nan_A=A","count_nan_B=B", "count_nan_C=C"]),
    cum_count_inf(cols=["count_inf_A=A","count_inf_B=B", "count_inf_C=C"]),
    cum_count_finite(cols=["count_finite_A=A","count_finite_B=B", "count_finite_C=C"]),
]

t_count = t.update_by(ops=op_arr)

[devinrsmith]

Did not take a look at anything except TableSpec stuff.

[devinrsmith]

"non-null", seems incorrect.

[devinrsmith]

Just pointing out (I think we do this in a couple other places), having values that are only applicable for some numeric types is a bit of a "code smell" - but something I guess we are "okay" with.

I'm worried about double logic; does ZERO include -0.0? Is -0.0 counted for negative?

Instead of an enum, I think we might want an EnumSet here where the user could say "I want to count NULL | NAN, or NULL | NAN | INFINITE, etc (and then, NON_NULL doesn't exist, it would just be represented by the lack of NULL).

If we really wanted to be more generally, we could even consider using a Filter/Expression - of course, we could limit it the scope of what would be allowable in this context, but it could give us room in the future to support more advance counts like "count number of odd longs", etc.

[devinrsmith]

I think having a method for per enum is not desirable... just expose enum (or, however it is configured) to the caller?

[lbooker42]

Performance Note

Extensive use of lambdas does not penalize performance in testing. Comparing to CumSum which has type-specific operators and evaluates every value and CumCountAll which does no evaluation.

Results show that the new operators match the CumSum performance in the bucketed tests (in which the partitioning cost dominates) and exceed it in zero-key tests.