Skip to content

Commit

Permalink
♻️ Refactor code for readability and consistency
Browse files Browse the repository at this point in the history
Several classes across various modules have been refactored to improve
readability by adding line breaks to long functions and making
indentation consistent. No functionality changes have been introduced.
In addition, unnecessary imports have been removed improving overall
cleanliness of the code.
  • Loading branch information
LeoColman committed Mar 5, 2024
1 parent 65a49e1 commit 1624aad
Show file tree
Hide file tree
Showing 32 changed files with 812 additions and 764 deletions.
9 changes: 9 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
root = true

[{*.kt,*.kts}]
charset = utf-8
end_of_line = lf
indent_size = 2
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true
26 changes: 18 additions & 8 deletions restaurants/README.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,46 @@
# Restaurants

This module is responsible for fetching restaurant items from USP (through [USP Restaurant API][1]) and persist them in a PostgresQL table, while also providing an API for the retrieval of these items.
This module is responsible for fetching restaurant items from USP (through [USP Restaurant API][1]) and persist them in
a PostgresQL table, while also providing an API for the retrieval of these items.

## Fetching from USP

We trust the [USP Restaurant API][1] definitions, and with them the [USPRestaurantItemRepository][3] is able to fetch items from USP's api.
We trust the [USP Restaurant API][1] definitions, and with them the [USPRestaurantItemRepository][3] is able to fetch
items from USP's api.

### Parsing the menu

It's important for this application to parse all values, as clients are going to expect a smarter output instead of just the menus (otherwise, why would this module be necessary?). For that we have the [MenuParsers][4]. We try with our best effort to parse the menus, however there is some difficulty as restaurants won't always report items in the same way.
It's important for this application to parse all values, as clients are going to expect a smarter output instead of just
the menus (otherwise, why would this module be necessary?). For that we have the [MenuParsers][4]. We try with our best
effort to parse the menus, however there is some difficulty as restaurants won't always report items in the same way.

A future point of improvement is being able to parse menus without such a brittle and forced solution.

## Saving to Postgres

Every item obtained from USP is piped into a PostgresQL item. This enables us to respond to clients even when USP's servers are down (which happen quite often).
Every item obtained from USP is piped into a PostgresQL item. This enables us to respond to clients even when USP's
servers are down (which happen quite often).

## Cache

When an item is successfully obtained, it's first persisted on the Postgres table (see above), and then kept in an in-memory cache. This cache allow us to quickly respond users while keeping costs down.
When an item is successfully obtained, it's first persisted on the Postgres table (see above), and then kept in an
in-memory cache. This cache allow us to quickly respond users while keeping costs down.

It's a good idea to use a cache here, as the data that clients want is usually the menus for the current week, making cache hits very likely.
It's a good idea to use a cache here, as the data that clients want is usually the menus for the current week, making
cache hits very likely.

## Restaurant Job

In order to have all records available - even for restaurants that aren't accessed regularly - we continuously try to get all restaurants from USP, using a fixed rate schedule in [RestaurantJob class][5].
In order to have all records available - even for restaurants that aren't accessed regularly - we continuously try to
get all restaurants from USP, using a fixed rate schedule in [RestaurantJob class][5].

This will both refresh our current cache and save any new menus to our persistent storage.



[1]: https://github.com/JopiterApp/USP-Restaurant-API

[3]: src/main/kotlin/repository/usp/USPRestaurantItemRepository.kt

[4]: src/main/kotlin/repository/usp/MenuParser.kt

[5]: src/main/kotlin/RestaurantJob.kt
3 changes: 1 addition & 2 deletions restaurants/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import org.jetbrains.kotlin.noarg.gradle.NoArgExtension

plugins {
kotlin("plugin.spring")
Expand Down Expand Up @@ -51,4 +50,4 @@ flyway {
url = System.getenv("JDBC_URL") ?: "jdbc:postgresql://localhost:5432/jopiter"
user = System.getenv("DATABASE_USER") ?: "jopiter"
password = System.getenv("DATABASE_PASS") ?: "password"
}
}
32 changes: 21 additions & 11 deletions restaurants/classifier/ClassifierDesign.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Classifier Design


### Temos:

Arquivos CSV onde a primeira coluna representa o nome do item e todas as outras colunas representam detalhes/categorias/classes dos itens.
Arquivos CSV onde a primeira coluna representa o nome do item e todas as outras colunas representam
detalhes/categorias/classes dos itens.

### Queremos

Expand All @@ -22,27 +22,37 @@ CSV:
João,Olhos Castanhos,Cabelo Preto

Tupla:

```
[nome = João, colunas=[Olhos Castanhos, Cabelo Preto]]
```


## Dados do Classificador

### Para treinar:
Receberá todas as tuplas de um determinado grupo. Todas são semelhantes entre si, e possuem a mesma quantidade de colunas. O classificador só será capaz de comparar tipos que ele já conhece

Receberá todas as tuplas de um determinado grupo. Todas são semelhantes entre si, e possuem a mesma quantidade de
colunas. O classificador só será capaz de comparar tipos que ele já conhece

### Para prever/classificar

Receberá apenas o nome e terá que deduzir as outras tuplas a partir disso

## Implementação do classificador

1. CSV é convertido para N-Tuplas
2. Tuplas passam por tratamento
1. corpus <= Transformar o nome com BagOfWords (lib smile kotlin nlp bag) passando a lista de stop words (TODO: melhorar lista) e um stemmer (FIXME, explicar melhor) de palavras em português
2. vocabulario <= todos os nomes das tuplas, após passado as stop words e o stemmer
3. corpusVetorizado <= Com o corpus (nomes como bag of words) e o vocabulário, contar quantas vezes cada vocábulo aparece em cada documento, armazenando cada documento
4. dadosProntosParaModelagem <= Usamos o algorítimo tfidf para melhorar a definição de importância de cada termo de cada documento (FIXME, explicar melhor)
1. corpus <= Transformar o nome com BagOfWords (lib smile kotlin nlp bag) passando a lista de stop words (TODO:
melhorar lista) e um stemmer (FIXME, explicar melhor) de palavras em português
2. vocabulario <= todos os nomes das tuplas, após passado as stop words e o stemmer
3. corpusVetorizado <= Com o corpus (nomes como bag of words) e o vocabulário, contar quantas vezes cada vocábulo
aparece em cada documento, armazenando cada documento
4. dadosProntosParaModelagem <= Usamos o algorítimo tfidf para melhorar a definição de importância de cada termo de
cada documento (FIXME, explicar melhor)
3. Com os dados prontos para modelagem, treinamos um modelo diferente para cada uma das N colunas:
1. Primeiro através de um algorítimo OVR (One Versus Rest) (FIXME explicar melhor OVR)
2. Este OVR será alimentado com os dados organizados em um vetor de suporte (support vector machine / SVM) (FIXME adicionar artigo que explicar porque svm é o melhor)
4. Estes modelos gerados compõem o clasificador, e podem ser exportados para armazenamento (economizando assim computação futura). É possível estimar pratos futuros neste classificador, tratando o novo documento da mesma forma que os antigos. Dessa forma temos as probabilidades que os modelos dão para cada uma das N colunas
1. Primeiro através de um algorítimo OVR (One Versus Rest) (FIXME explicar melhor OVR)
2. Este OVR será alimentado com os dados organizados em um vetor de suporte (support vector machine / SVM) (FIXME
adicionar artigo que explicar porque svm é o melhor)
4. Estes modelos gerados compõem o clasificador, e podem ser exportados para armazenamento (economizando assim
computação futura). É possível estimar pratos futuros neste classificador, tratando o novo documento da mesma forma
que os antigos. Dessa forma temos as probabilidades que os modelos dão para cada uma das N colunas
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
package app.jopiter.restaurants.classifier

import kotlin.streams.toList

@Suppress("RECEIVER_NULLABILITY_MISMATCH_BASED_ON_JAVA_ANNOTATIONS")
class StopwordsProvider {

Expand All @@ -10,4 +8,5 @@ class StopwordsProvider {
}
}

private fun portugueseStopwordsFile() = StopwordsProvider::class.java.classLoader.getResourceAsStream("portuguese_stopwords.txt")!!
private fun portugueseStopwordsFile() =
StopwordsProvider::class.java.classLoader.getResourceAsStream("portuguese_stopwords.txt")!!
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@

package app.jopiter.restaurants.classifier

import app.jopiter.restaurants.classifier.ClassifiableRow
import app.jopiter.restaurants.classifier.Classifier
import io.kotest.core.spec.style.FunSpec
import io.kotest.core.spec.style.funSpec
import io.kotest.matchers.doubles.shouldBeGreaterThanOrEqual
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ package app.jopiter.restaurants
import app.jopiter.restaurants.classifier.RestaurantItemClassifier
import app.jopiter.restaurants.model.Campus
import app.jopiter.restaurants.model.ClassifiedRestaurantItem
import app.jopiter.restaurants.model.RestaurantItem
import app.jopiter.restaurants.repository.RestaurantItemRepository
import io.swagger.v3.oas.annotations.Operation
import io.swagger.v3.oas.annotations.Parameter
Expand All @@ -40,48 +39,56 @@ import java.time.LocalDate
@RestController
@RequestMapping("\${api.base.path}/restaurants")
class RestaurantController(
private val restaurantItemRepository: RestaurantItemRepository,
private val restaurantItemClassifier: RestaurantItemClassifier
private val restaurantItemRepository: RestaurantItemRepository,
private val restaurantItemClassifier: RestaurantItemClassifier
) {

@Operation(
summary = "List all restaurants and their campi",
description = "List all restaurants that are available, including to which campus it belongs",
tags = ["restaurant"],
responses = [
ApiResponse(responseCode = "200", content = [
Content(array = ArraySchema(schema = Schema(implementation = Campus::class)),
mediaType = "application/json")
])
@Operation(
summary = "List all restaurants and their campi",
description = "List all restaurants that are available, including to which campus it belongs",
tags = ["restaurant"],
responses = [
ApiResponse(
responseCode = "200", content = [
Content(
array = ArraySchema(schema = Schema(implementation = Campus::class)),
mediaType = "application/json"
)
]
)
@GetMapping
fun list() = Campus.values().toList()
)
]
)
@GetMapping
fun list() = Campus.values().toList()

@Operation(
summary = "List Items",
description = "Retrieves all items for the chosen dates and restaurant",
tags = ["restaurant"],
@Operation(
summary = "List Items",
description = "Retrieves all items for the chosen dates and restaurant",
tags = ["restaurant"],

parameters = [
Parameter(
name = "restaurantId", description = "The restaurant ID as defined by /restaurants", required = true
),
Parameter(
name = "date", description = "The dates you want to fetch items for. ISO_LOCAL_DATE format (yyyy-MM-dd)"
)
],
parameters = [
Parameter(
name = "restaurantId", description = "The restaurant ID as defined by /restaurants", required = true
),
Parameter(
name = "date", description = "The dates you want to fetch items for. ISO_LOCAL_DATE format (yyyy-MM-dd)"
)
],

responses = [
ApiResponse(responseCode = "200", content = [
Content(array = ArraySchema(schema = Schema(implementation = ClassifiedRestaurantItem::class)),
mediaType = "application/json")
])
responses = [
ApiResponse(
responseCode = "200", content = [
Content(
array = ArraySchema(schema = Schema(implementation = ClassifiedRestaurantItem::class)),
mediaType = "application/json"
)
]
)
@GetMapping("/items")
fun items(
@RequestParam("restaurantId") restaurantId: Int,
@RequestParam("date") @DateTimeFormat(iso = DATE) dates: Set<LocalDate>,
) = restaurantItemRepository.get(restaurantId, dates).map(restaurantItemClassifier::classify)
)
]
)
@GetMapping("/items")
fun items(
@RequestParam("restaurantId") restaurantId: Int,
@RequestParam("date") @DateTimeFormat(iso = DATE) dates: Set<LocalDate>,
) = restaurantItemRepository.get(restaurantId, dates).map(restaurantItemClassifier::classify)
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,18 @@ import app.jopiter.restaurants.repository.RestaurantItemRepository
import org.slf4j.LoggerFactory
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Component
import java.time.LocalDate.now
import java.util.concurrent.ScheduledExecutorService
import java.util.concurrent.TimeUnit.MINUTES

@Component
class RestaurantJob(
private val repository: RestaurantItemRepository
private val repository: RestaurantItemRepository
) {

private val logger = LoggerFactory.getLogger(this::class.java)
private val logger = LoggerFactory.getLogger(this::class.java)

@Scheduled(fixedRate = 15_000)
fun run() = Restaurant.entries.forEach {
logger.info("Starting scheduled execution to fetch $it")
repository.fetchFromUsp(it.id)
logger.info("Finished scheduled execution to fetch $it")
}
@Scheduled(fixedRate = 15_000)
fun run() = Restaurant.entries.forEach {
logger.info("Starting scheduled execution to fetch $it")
repository.fetchFromUsp(it.id)
logger.info("Finished scheduled execution to fetch $it")
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import app.jopiter.restaurants.model.DessertFoodGroup
import app.jopiter.restaurants.model.DessertItem
import app.jopiter.restaurants.model.DessertPreparation
import app.jopiter.restaurants.model.ProteinFoodGroup
import app.jopiter.restaurants.model.ProteinPreparation
import app.jopiter.restaurants.model.ProteinItem
import app.jopiter.restaurants.model.ProteinPreparation
import app.jopiter.restaurants.model.RestaurantItem
import app.jopiter.restaurants.model.VegetarianFoodGroup
import app.jopiter.restaurants.model.VegetarianItem
Expand Down Expand Up @@ -45,17 +45,29 @@ class RestaurantItemClassifier(

fun classifyProtein(item: String): ProteinItem {
val classification = proteinClassifier.classify(item)
return ProteinItem(item, ProteinFoodGroup.find(classification.foodGroup), ProteinPreparation.find(classification.preparation))
return ProteinItem(
item,
ProteinFoodGroup.find(classification.foodGroup),
ProteinPreparation.find(classification.preparation)
)
}

fun classifyVegetarian(item: String): VegetarianItem {
val classification = vegetarianClassifier.classify(item)
return VegetarianItem(item, VegetarianFoodGroup.find(classification.foodGroup), VegetarianPreparation.find(classification.preparation))
return VegetarianItem(
item,
VegetarianFoodGroup.find(classification.foodGroup),
VegetarianPreparation.find(classification.preparation)
)
}

fun classifyDessert(item: String): DessertItem {
val classification = dessertClassifier.classify(item)
return DessertItem(item, DessertFoodGroup.find(classification.foodGroup), DessertPreparation.find(classification.preparation))
return DessertItem(
item,
DessertFoodGroup.find(classification.foodGroup),
DessertPreparation.find(classification.preparation)
)
}

}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package app.jopiter.restaurants.model

import com.fasterxml.jackson.annotation.JsonFormat
import org.springframework.stereotype.Component
import java.time.LocalDate


Expand Down
Loading

0 comments on commit 1624aad

Please sign in to comment.