feat: s3 data forwarding

[imranismail]

Support use-case to deploy S3 data forwarding destination and rules

resource "aws_s3_bucket" "this" {
  bucket = "${terraform.workspace}-sumologic"
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::926226587429:root"]
    }

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = ["au:0000000000595DED"]
    }
  }
}

resource "aws_iam_role" "this" {
  name               = "${terraform.workspace}-sumologic-s3-writer"
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

data "aws_iam_policy_document" "role" {
  statement {
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.this.arn}/*"]
  }
}

resource "aws_iam_role_policy" "this" {
  name   = "sumologic-s3-writer"
  role   = aws_iam_role.this.id
  policy = data.aws_iam_policy_document.role.json
}

data "sumologic_partitions" "this" {}

resource "sumologic_s3_data_forwarding_destination" "this" {
  depends_on          = [aws_iam_role_policy.this]
  name                = "${terraform.workspace}-sumologic"
  bucket_name         = aws_s3_bucket.this.bucket
  authentication_mode = "RoleBased"
  role_arn            = aws_iam_role.this.arn
  encrypted           = true
}


locals {
  partitions = {
    for partition in data.sumologic_partitions.this.partitions : partition.name => partition
    if startswith(partition.name, "${terraform.workspace}_eks_") && partition.analytics_tier != "infrequent"
  }
}


resource "sumologic_s3_data_forwarding_rule" "this" {
  for_each       = local.partitions
  index_id       = each.value.id
  destination_id = sumologic_s3_data_forwarding_destination.this.id
  file_format    = "{index}/{day}/{hour}_{minute}_{second}"
}

[sumovishal]

@vsinghal13 can you take a look?

[vsinghal13]

@sumovishal this is not owned by Data Collection team.

[vsinghal13]

@ksbagr seems to be last person who made some changes to partitions. can you review this?

[imranismail]

Hi folks, any chance to get this in?

[imranismail]

Hey folks, any chance to get this in or do we have to maintain a fork of this from now on?

[sumovishal]

@AyanGhatak can you take a look?

[ssharma-sumologic]

Instead of fetching default no of partitions, it might be better to fetch max limit allowed 1000 to reduce number of api calls https://api.sumologic.com/docs/#operation/listPartitions

[ssharma-sumologic]

This should be RuleRead not update

[ssharma-sumologic]

Can you please rename this to DestinationName for clarity

[ssharma-sumologic]

Same comment here to rename this to destination_name to have similar naming convention with api

[ssharma-sumologic]

This should be Read not Update

[ssharma-sumologic]

This shouldn't be changed, right @sumovishal

[ssharma-sumologic]

Tests are missing

[sumovishal]

@ssharma-sumologic please review again. If the PR looks good, can you open it against the provider repo? We can't run acceptance tests here.

[ssharma-sumologic]

please change this to d.index_id

[ssharma-sumologic]

The id of a data forwarding rule isn't equal to the indexId. Please set the id of the rule here

[ssharma-sumologic]

I believe there's some confusion around Id and indexId. IndexId is the id of the partition/view , while the id is a unique identifier of the rule. Please see the docs https://api.sumologic.com/docs/#operation/createDataForwardingRule for more reference

[namangoya]

@imranismail - We will be releasing the changes here.