New Loss Format

[hfaghihi15]

Hi all,

I think we need to implement a better loss function setting where each learner also receives the loss function attached to it. But apart from that we also need to be able to attach masking to the losses.

I think the best way to do so is to define mask property on the LabelReaderSensor. Let me know what you think?

The system should contain, masking, weight integration, different and custom loss functions, applying different loss functions during training, pertaining, pipelines, and so on.

[hfaghihi15]

@AlexWan0 could you explain what you need for weighting and your possible idea on where is the best place to put that information in the pipeline of DomiKnows programming?

[guoquan]

Please also refer to this poi program implementation with which we try to attach loss with learners.

DomiKnowS/regr/program/model/pytorch.py

Lines 275 to 332 in 651c7dd

	class PoiModelToWorkWithLearnerWithLoss(TorchModel):
	def __init__(self, graph, poi=None):
	super().__init__(graph)
	if poi is not None:
	self.poi = poi
	else:
	self.poi = self.default_poi()
	self.loss_tracker = MacroAverageTracker()
	self.metric_tracker = None
	def reset(self):
	if self.loss_tracker is not None:
	self.loss_tracker.reset()
	if self.metric_tracker is not None:
	self.metric_tracker.reset()
	def default_poi(self):
	poi = []
	for prop in self.graph.get_properties():
	if len(list(prop.find(TorchSensor))) > 1:
	poi.append(prop)
	return poi
	def populate(self, builder):
	losses = {}
	metrics = {}
	for prop in self.poi:
	targets = []
	predictors = []
	for sensor in prop.find(TorchSensor):
	sensor(builder)
	if sensor.label:
	targets.append(sensor)
	else:
	predictors.append(sensor)
	for predictor in predictors:
	# TODO: any loss or metric or genaral function apply to just prediction?
	pass
	for target, predictor in product(targets, predictors):
	if predictor._loss is not None:
	losses[predictor, target] = predictor.loss(builder, target)
	if predictor._metric is not None:
	metrics[predictor, target] = predictor.metric(builder, target)
	self.loss_tracker.append(losses)
	# self.metrics_tracker.append(metrics)
	loss = sum(losses.values())
	return loss, metrics
	@property
	def loss(self):
	return self.loss_tracker
	@property
	def metric(self):
	# return self.metrics_tracker
	pass

[AlexWan0]

For the Typenet example, there is a large class imbalance towards the negative classes, as there's only a couple positive cases out of ~1000 total classes per mention.
I feel that a convenient place to put the class weights might be in the reader, as that's probably where I would be calculating the class ratios. Also, I think the Typenet paper calculates their class weights per batch, so the reader might be the only way to pass those values in.
see https://github.com/dair-iitd/dl-with-constraints/blob/master/typenet/src/deploy_mil_constraints.py
line 302

[hfaghihi15]

Please also refer to this poi program implementation with which we try to attach loss with learners.

DomiKnowS/regr/program/model/pytorch.py

Lines 275 to 332 in 651c7dd

	class PoiModelToWorkWithLearnerWithLoss(TorchModel):
	def __init__(self, graph, poi=None):
	super().__init__(graph)
	if poi is not None:
	self.poi = poi
	else:
	self.poi = self.default_poi()
	self.loss_tracker = MacroAverageTracker()
	self.metric_tracker = None
	def reset(self):
	if self.loss_tracker is not None:
	self.loss_tracker.reset()
	if self.metric_tracker is not None:
	self.metric_tracker.reset()
	def default_poi(self):
	poi = []
	for prop in self.graph.get_properties():
	if len(list(prop.find(TorchSensor))) > 1:
	poi.append(prop)
	return poi
	def populate(self, builder):
	losses = {}
	metrics = {}
	for prop in self.poi:
	targets = []
	predictors = []
	for sensor in prop.find(TorchSensor):
	sensor(builder)
	if sensor.label:
	targets.append(sensor)
	else:
	predictors.append(sensor)
	for predictor in predictors:
	# TODO: any loss or metric or genaral function apply to just prediction?
	pass
	for target, predictor in product(targets, predictors):
	if predictor._loss is not None:
	losses[predictor, target] = predictor.loss(builder, target)
	if predictor._metric is not None:
	metrics[predictor, target] = predictor.metric(builder, target)
	self.loss_tracker.append(losses)
	# self.metrics_tracker.append(metrics)
	loss = sum(losses.values())
	return loss, metrics
	@property
	def loss(self):
	return self.loss_tracker
	@property
	def metric(self):
	# return self.metrics_tracker
	pass

Thanks for the hint.
I was looking at the code but I think this wouldn't probably work with the new changes. I was also thinking whether it would be better to attach it to the readerSensors which read labels, the ModuleLearners, or even at the program instance by defining a dictionary that maps the properties to their appropriate loss function.
What do you think?