Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/elastic search pull request #181

Merged
merged 95 commits into from
Nov 28, 2023
Merged

feature/elastic search pull request #181

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
1c9bbae
First approach to use the elasticsearch service
Aug 14, 2023
eb73469
elasticsearch query example
AdrianAlcolea Aug 14, 2023
8bd8115
elasticsearch query complete example
AdrianAlcolea Aug 14, 2023
2e7934b
Elasticsearch setup configured
AdrianAlcolea Aug 16, 2023
5ba4a7c
Remove .DS_Store and add it to dockerignore
AdrianAlcolea Aug 16, 2023
fcdc19c
Merge branch 'develop' of https://github.com/aiondemand/AIOD-rest-api…
josvandervelde Aug 17, 2023
22e8577
Merge branch 'develop' of https://github.com/aiondemand/AIOD-rest-api…
josvandervelde Aug 23, 2023
6d665bb
Created an elastic search endpoint. It isnt working yet
josvandervelde Aug 23, 2023
1123335
Just to test the new database
AdrianAlcolea Aug 23, 2023
80b9c3c
To rebase develop
AdrianAlcolea Aug 23, 2023
30d6eda
Working but not up-to-date with develop branch
AdrianAlcolea Aug 23, 2023
28e58fc
Service working. Need to get up-to-date with develop
AdrianAlcolea Aug 23, 2023
a93fa89
Resolved merge conflicts with develop
josvandervelde Aug 24, 2023
f2aa463
Some bugfixes for the connectors
josvandervelde Aug 24, 2023
f2893f4
Deleting docker images in clean script
josvandervelde Aug 24, 2023
3d17c8c
Fixed issues with authentication
josvandervelde Aug 24, 2023
575faaf
Publication search seems to work. TODO: test cases and other resourcse
josvandervelde Aug 24, 2023
afdb897
platform and platform_identifier changed from aiod_entry to each inst…
AdrianAlcolea Aug 24, 2023
895a8d2
Created testcase for publication search
josvandervelde Aug 24, 2023
4a2e0f9
Made ElasticSearch router generic, implemented it for dataset
josvandervelde Aug 24, 2023
d6ce8e9
Logstash configured for dataset, experiment, ml_model, publication, a…
AdrianAlcolea Aug 24, 2023
5fb737b
Logstash configured for dataset, experiment, ml_model, publication, a…
AdrianAlcolea Aug 24, 2023
0275144
Logstash waits until fill-db-with-examples ends
AdrianAlcolea Aug 24, 2023
ccc1ffd
Resolved merge conflicts with develop
josvandervelde Aug 24, 2023
5de434c
take src from develop
AdrianAlcolea Aug 30, 2023
5ff4498
Copied entire develop branch
AdrianAlcolea Aug 30, 2023
cae4321
Logstash configuration readapted to new names
AdrianAlcolea Aug 30, 2023
80aef7d
Logstash configuration readapted to new names
AdrianAlcolea Aug 30, 2023
ca2f44c
Merged with develop
AdrianAlcolea Aug 30, 2023
69ffa34
added ai4experiments to platform names
AdrianAlcolea Aug 31, 2023
ba49bbb
Copied initial search routers to start creating them
AdrianAlcolea Sep 5, 2023
ce08b0a
Examples of ml_model, dataset and experiment used to insert ai4experi…
AdrianAlcolea Sep 6, 2023
84686ea
Descriptions of the ai4experiment data improved
AdrianAlcolea Sep 6, 2023
7802496
platform added to mappings
AdrianAlcolea Sep 13, 2023
fdee2b7
elasticsearch query example completed
AdrianAlcolea Sep 13, 2023
2c3e9d7
First version of search service working
AdrianAlcolea Sep 13, 2023
0280c26
Search router tests implemented
AdrianAlcolea Sep 14, 2023
106d8c0
Merge with develop
AdrianAlcolea Sep 27, 2023
a0cc086
Search fields selection added
AdrianAlcolea Sep 28, 2023
97b361c
Added search for event, news, ortganisation and project
AdrianAlcolea Oct 10, 2023
e69620c
Added routers for event, news, organisation and project
AdrianAlcolea Oct 17, 2023
511b779
merged with develop
AdrianAlcolea Oct 17, 2023
ee4858c
merged with develop
AdrianAlcolea Oct 23, 2023
3e5d5f4
Logstash names changed
AdrianAlcolea Oct 23, 2023
2146a3d
added logstash_config.py, just for having it there
AdrianAlcolea Oct 26, 2023
f964457
Pagination changed to actual pages
AdrianAlcolea Oct 26, 2023
0fe2fe1
Pagination changed to actual pages
AdrianAlcolea Oct 26, 2023
464c270
Application areas added to elasticsearch resuts
AdrianAlcolea Oct 31, 2023
8289131
merged with develop
AdrianAlcolea Nov 6, 2023
85ba925
First version with deletion
AdrianAlcolea Nov 8, 2023
cbe2a82
merged with develop
AdrianAlcolea Nov 8, 2023
0865424
Prepared to be merged with develop
AdrianAlcolea Nov 9, 2023
518036c
merged with develop
AdrianAlcolea Nov 9, 2023
92038e7
pull request modifications
AdrianAlcolea Nov 10, 2023
9b1fd52
pull request modifications
AdrianAlcolea Nov 11, 2023
f1df999
merged with develop
AdrianAlcolea Nov 11, 2023
74ac4f0
pull request modifications
AdrianAlcolea Nov 11, 2023
7c20576
Combined search with sql queries in process
AdrianAlcolea Nov 13, 2023
88150dd
merged with develop
AdrianAlcolea Nov 13, 2023
0aeaf25
Search functionality combined with optional SQL statment to retrieve …
AdrianAlcolea Nov 13, 2023
05c6814
Elasticsearch and logstash configuration integrated in src
AdrianAlcolea Nov 15, 2023
a96e9f6
merged with develop
AdrianAlcolea Nov 15, 2023
92b05cf
Search router tests actualised
AdrianAlcolea Nov 15, 2023
22d8df2
Search router tests actualised
AdrianAlcolea Nov 15, 2023
03868c3
Search router tests actualised
AdrianAlcolea Nov 15, 2023
e9f14be
pre-commit passed
AdrianAlcolea Nov 20, 2023
4821f67
All test passed and working. Not merged with develop
AdrianAlcolea Nov 20, 2023
8c65397
huggingface connector test to its original state
AdrianAlcolea Nov 20, 2023
120f97a
back to commented huggingface connector
AdrianAlcolea Nov 20, 2023
3e5c446
Fixing unittests by making sure Elasticsearch instance can also be cr…
josvandervelde Nov 20, 2023
9bdd974
Merge branch 'develop' of https://github.com/aiondemand/AIOD-rest-api…
josvandervelde Nov 20, 2023
3daef84
clean logstash configuration
AdrianAlcolea Nov 20, 2023
7e0861a
clean logstash configuration
AdrianAlcolea Nov 20, 2023
649325d
clean logstash configuration
AdrianAlcolea Nov 20, 2023
f4739e9
clean logstash configuration
AdrianAlcolea Nov 20, 2023
985949d
clean logstash configuration
AdrianAlcolea Nov 20, 2023
ae2ac9e
clean logstash configuration
AdrianAlcolea Nov 20, 2023
cc8c22f
logstash config files generated with jinja2
AdrianAlcolea Nov 24, 2023
b73c011
logstash config files generated with jinja2
AdrianAlcolea Nov 24, 2023
6ae492d
Logstash config files generated with jinja2. All test passed, but not…
AdrianAlcolea Nov 24, 2023
eaf3451
Resolved merge conflicts with develop
josvandervelde Nov 24, 2023
ca5c137
Second round of pull request comments
AdrianAlcolea Nov 25, 2023
4f17532
Second round of pull request comments
AdrianAlcolea Nov 25, 2023
b60c9aa
Second round of pull request comments
AdrianAlcolea Nov 25, 2023
7c2106f
Second round of pull request comments
AdrianAlcolea Nov 25, 2023
0e13852
Second round of pull request comments
AdrianAlcolea Nov 25, 2023
453b1cd
Created data/elasticsearch/.gitkeep to make sure it exists with the r…
josvandervelde Nov 27, 2023
6767136
Deleted autogenerated file logstash/config/logstash.yml
josvandervelde Nov 27, 2023
defc100
cleanup
josvandervelde Nov 27, 2023
5f79f0d
Making sure docker compose up works even if generated files do not ex…
josvandervelde Nov 27, 2023
58b2514
Make sure data folders are always created with correct permissions (t…
josvandervelde Nov 27, 2023
0d6bd3b
Added default logstash configuration
josvandervelde Nov 27, 2023
82b049a
Fixed docker compose
josvandervelde Nov 27, 2023
85c76b5
Using FastAPI input validation
josvandervelde Nov 27, 2023
3adc54b
Made status nullable, so that we can return an empty status in the se…
josvandervelde Nov 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ services:
- ./logstash:/logstash
command: >
/bin/bash -c "python setup/logstash/generate_logstash_config_files.py &&
python setup/elasticsearch/generate_elasticsearch_indices.py"
python setup/es/generate_elasticsearch_indices.py"
restart: "no"
depends_on:
elasticsearch:
Expand Down
9 changes: 1 addition & 8 deletions logstash/config/logstash.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1 @@
# This file has been generated by `generate_logstash_config.py`
# file, placed in `src/setup/logstash`
# -------------------------------------------------------------
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme

201 changes: 0 additions & 201 deletions logstash/pipeline/conf/init_table.conf
View file
Open in desktop

This file was deleted.

Loading
Loading