Different tilt values in TX files and in STM files

[BaptisteVandecrux]

I kept seeing slightly different radiation values when replacing old transmission files by raw logger files and eventually tracked it down.

In TX files, tilt is available every 6 hours while in RAW files it's every hour (at least).

In the L0toL1, NaNs in tilt_x and tilt_y are interpolated:

pypromice/src/pypromice/process/L0toL1.py

Lines 304 to 338 in 3813c77

	def smoothTilt(tilt, win_size):
	'''Smooth tilt values using a rolling window. This is translated from the
	previous IDL/GDL smoothing algorithm:
	tiltX = smooth(tiltX,7,/EDGE_MIRROR,MISSING=-999) & tiltY = smooth(tiltY,7,/EDGE_MIRROR, MISSING=-999)
	endif
	In Python, this should be
	dstxy = dstxy.rolling(time=7, win_type='boxcar', center=True).mean()
	But the EDGE_MIRROR makes it a bit more complicated
	Parameters
	----------
	tilt : xarray.DataArray
	Array (either 'tilt_x' or 'tilt_y'), tilt values (can be in degrees or voltage)
	win_size : int
	Window size to use in pandas 'rolling' method.
	e.g. a value of 7 spans 70 minutes using 10 minute data.
	Returns
	-------
	tdf_rolling : tuple, as: (str, numpy.ndarray)
	The numpy array is the tilt values, smoothed with a rolling mean
	'''
	s = int(win_size/2)
	tdf = tilt.to_dataframe()
	mirror_start = tdf.iloc[:s][::-1]
	mirror_end = tdf.iloc[-s:][::-1]
	mirrored_tdf = pd.concat([mirror_start, tdf, mirror_end])
	tdf_rolling = (
	('time'),
	mirrored_tdf.rolling(
	win_size, win_type='boxcar', min_periods=1, center=True
	).mean()[s:-s].values.flatten()
	)
	return tdf_rolling

First of all, such filtering and interpolation should not be done at L0toL1 level, but rather on the L1toL2 level.

But secondly, when done on (noisy) 6-hourly TX data, this interpolation adds spurious "waves" in the tilt data. And although there is an attempt in L1toL2 to filter noise, the result down the line is different when the input data is from a TX file and a RAW file.

Here is an example at KAN_U in October 2023 for which we have both old transmission and logger data:

• We should make sure that at L0toL1, the data is not modified, just made readable.
• In L1toL2, we should make sure that the smoothing and interpolation lead to similar values when calculated on 6hr and 1hr data.
• We need an assessment of the tilt sensor: is the variations seen in the hourly RAW data true ? or should it be filtered out?