support human shape annotation in meta column info

[yupbank]

data = [Row(x=[float(x), float(2 * x)], key=str(x % 2)) for x in range(1, 6)]
df_analyzed = tfs.analyze(sqlContext.createDataFrame(data))
df = sqlContext.createDataFrame(data)

# this would work
grouped_df_analyzed = df_analyzed.groupby('key')

In [79]: with tf.Graph().as_default() as g:
    ...:     x = tf.placeholder(tf.double, [None, 2], name='x_input')
    ...:     y = tf.reduce_mean(x, 0, name='x')
    ...:     df_1 = tfs.aggregate(y, grouped_df_analyzed)

#while this would fail..

grouped_df  = df.groupby('key')

In [80]: with tf.Graph().as_default() as g:
    ...:     x = tf.placeholder(tf.double, [None, 2], name='x_input')
    ...:     y = tf.reduce_mean(x, 0, name='x')
    ...:     df_1 = tfs.aggregate(y, grouped_df)

# with reason

java.lang.Exception: The data column 'x' has shape [?,?], not compatible with shape [?,2] requested by the TF graph

which would be useful if there is a function the take human input into the meta data...

[yupbank]

@tjhunter would builder or experiment operator a good place to append human annotation?

[thunterdb]

Hello Peng, yes you need to analyze the dataframe first, unless it contains some simple (scalar) types. This may be relaxed in the future.

On January 23, 2018 at 6:18:34 PM, Peng Yu ***@***.***) wrote: turns out it can be done, but i need to `tfs.analyze(df)` first...

wondering if it is possible

to give it to df instead of analysis it -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

#137 (comment)

[thunterdb]

Hello Peng, what do you mean by human annotation?

On January 23, 2018 at 7:18:05 PM, Peng Yu ***@***.***) wrote: @tjhunter would builder or experiment operator a good place to append

human annotation?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

#137 (comment)

[yupbank]

Hi, I was referring to whether we should provide a interface let user pass in the shape of the dataframe columns. In my example, if I can add shape=(None,2) to column x's meta data, I don't need to analyze the whole dataframe, which have expensive cost

[thunterdb]

@yupbank sorry I missed your response. There is a workaround documented there that should do the trick for your use case:
https://groups.google.com/forum/#!topic/tensorframes/g3Dm97oFvVw
It would be great to add this functionality into tensorframes. If it works for you, you should feel free to open a pull request.