Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow running on remote orchestrator #80

Merged
merged 5 commits into from
Dec 7, 2023
+39 −11
Merged

Allow running on remote orchestrator #80

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7a6600b
Allow running on remote orchestrator
AlexejPenner Oct 30, 2023
7acd938
Update zenml version in requirements
AlexejPenner Oct 30, 2023
47df80c
Move secrets to zenml secrets store
AlexejPenner Nov 2, 2023
16d9cfc
Removed explicit API_KEY as env var in docker
AlexejPenner Nov 2, 2023
1a24887
Docker settings set to download files
AlexejPenner Nov 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion langchain-llamaindex-slackbot/src/local_testing_slackbot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ def reply_in_thread(body: dict, say, context):
thread_ts = event.get("thread_ts", None) or event["ts"]

if context["bot_user_id"] in event["text"]:
logger.debug(f"Received message: {event['text']}")
logger.info(f"Received message: {event['text']}")
if event.get("thread_ts", None):
full_thread = [
f"{msg['text']}"
Expand Down Expand Up @@ -107,6 +107,7 @@ def reply_in_thread(body: dict, say, context):
question=event["text"],
verbose=True,
)
logger.info(output)
say(text=output, thread_ts=thread_ts)


Expand Down
23 changes: 20 additions & 3 deletions langchain-llamaindex-slackbot/src/pipelines/index_builder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,34 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing
# permissions and limitations under the License.

import os

from steps.index_generator import index_generator
from steps.url_scraper import url_scraper
from steps.web_url_loader import web_url_loader
from zenml import pipeline
from zenml.config import DockerSettings

pipeline_name = "zenml_docs_index_generation"
docker_settings = DockerSettings(
requirements=[
"langchain==0.0.263",
"openai==0.27.2",
"slack-bolt==1.16.2",
"slack-sdk==3.20.0",
"fastapi",
"flask",
"uvicorn",
"gcsfs==2023.5.0",
"faiss-cpu==1.7.3",
"unstructured==0.5.7",
"tiktoken",
"bs4"
],
Comment on lines +25 to +38
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't tell if this is different from the fixed files themselves, but would it be possible instead to just specify the requirements file itself? Seems safer, given that these files do actually get updated fairly often.

docker_settings = DockerSettings(requirements="/path/to/requirements.txt")

environment={"OPENAI_API_KEY": os.getenv("OPENAI_API_KEY")}
)


@pipeline(name=pipeline_name)
@pipeline(name=pipeline_name, settings={"docker": docker_settings})
def docs_to_index_pipeline(
docs_url: str = "",
repo_url: str = "",
Expand Down
2 changes: 1 addition & 1 deletion langchain-llamaindex-slackbot/src/requirements-slackbot.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ langchain==0.0.263
openai==0.27.2
slack-bolt==1.16.2
slack-sdk==3.20.0
zenml[connectors-gcp]==0.45.3
zenml[connectors-gcp]==0.45.5
fastapi
flask
uvicorn
Expand Down
2 changes: 1 addition & 1 deletion langchain-llamaindex-slackbot/src/requirements-zenml-io-qa.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ langchain>=0.0.125,<=0.0.263
openai>=0.27.2,<=0.27.8
slack-bolt==1.16.2
slack-sdk==3.20.0
zenml==0.44.1
zenml==0.45.5
fastapi
flask
uvicorn
Expand Down
Loading