notes.org

General Knowledge

Setting up postgres

docker run --name postgres \
  -e POSTGRES_PASSWORD=password \
  -d postgres

Setting up pgadmin via docker

docker run -i -t --rm \
               -p 80:80 \ # map HOST:CONTAINER ports
               -e '[email protected]' \
               -e 'PGADMIN_DEFAULT_PASSWORD=password'
               -v  \
               -d dpage/pgadmin4

Communicating between containers

• By default, containers are hooked into the bridge docker subnetwork.
• Running the following command will list out the containers connected to bridge

  docker network inspect bridge    
      

• Containers can then access each other by using the IP address assigned to them in the docker subnetwork
• This is useful for connecting from pgadmin to a postgres

Hiccups

Followers/Following should be returned by actor endpoint

Everything should have an id, else things will break

Accept headers should be specified when looking up elements, else things will fail

Overview

consists of two layers:

• server to server federation
• client to server protocol

representation of users, with properties:

inbox

messages from the world

outbox

send messages out

these endpoints are listed in the activitystreams description:

{"@context": "https://www.w3.org/ns/activitystreams",
  "type": "Person",
  "id": "https://social.example/alyssa/",
  "preferredUsername": "alyssa",
  "summary": "Lisp enthusiast",
  "inbox": "https://social.example/alyssa/inbox/",
  "outbox": "https://social.example/alyssa/outbox/"
}

         read latest                     send messages
          messages                       to actor from 
                                             server
actor    <---GET---      [inbox]         <---POST---    rest of the world

actor    ---POST--->     [outbox]         ---GET--->    rest of the world
        post messages                   retreive messages
          to world                      produced by server

• example usecase:
  • alyssa’s client posts a message to her outbox on her server
    • NOTE: as a message is not an activity, her server wraps it in a create object
  • server looks up ben’s activitystreams actor and posts object to inbox
  • ....
  • alysa’s client gets her inbox, and recieves ben’s message (has also been wrapped in create)
• another usecase:
  • alyssa wants to like ben’s post
  • alyssa’s client posts to outbox {type: like, to=[“ben”]; actor: alyssa; object: “id of ben’s post”}
    • NOTE: as a like is an activity, no need to wrap in create
• to send a public/followers message, include followers & public

Parts I need to implement

Inbox

GET - returns messages from user

POST - adds a message to user

Outbox

GET - returns outbound messages for user

POST - adds outbound message for user

Microblog.pub

Files

Core

Activity pub

_meta

save (box, activity)

• saves an activity to a box
• logic is:
  • retrieve meta information (lookup what _meta does)
  • if into outbox and is follow, then set follow status of meta information to waiting
  • else if create activity,
    • get object being created
    • retrieve mentions and hashtags
    • append to metadata
  • insert into box, raw activity, type, id, and meta information

outbox is blocked

• looks up in db if there exists a entry
  • in outbox
  • with block status
  • id being the selected id
  • and in metadata, has not been undone

activity url

• returns the BASE_URL plus, url_for (outbox_detail, id = item_id)

post to inbox (activity)

• logic is:
  • get actor for activity
  • if activity with remote_id matching activity in inbox, ignore
  • else
    • save activity to inbox
    • if not a delete or update activity
      • queue task to update cached actor
    • queue task to process activity
    • queue task to finish post to inbox

impl-specific logic

• the following logic is run before any of the generic processing
  • if outbox is blocked, return without doing anything
  • else if
    • actor is an application type
    • actor id ends with /relay
    • activity type is announce
    • and there are no creation activities with the same id
    • and no replies from the remote id of object id of activity
    • then queue a process reply of object id
  • else if
    • update activity
    • id is object_id
    • then queue task to update cached actor

save reply (activity, meta)

• get visibility of activity
• get published date of activity (or now if not present)
• insert activity into replies

  { "activity": "<json>", "type": activity.type, "remote_id": activity.id, 
    "meta": {
       "undo": false, "deleted": false, "public": is_public, 
       "server": urlparse(id).hostname, "visibility", "actor_id": activity.get_actor().id
       "published"
    }
  }   
      

post to outbox (activity)

• if activity is create type, wrap activity in create type
• obj_id = assign create a random id
• create uri for obj id
• if create activity,
  • activity._data.object.id = base_url + url_for outbox + obj_id
  • if no url field in data for object, then assign object url: base_url + url_for note_by_id + obj_id
  • reset object cache for activity
• save object to outbox
• queue task to update the cached actor
• queue task to finish posting to outbox activity

accept follow (activity)

• retrieve id for actor
• create accept object, with type follow, id being activity.id, actor being actor_id, send to actor, published now
• update one_activity by the remote id and set accepted status to true
• post activity to outbox accepted

handle question reply (create, question)

• retrieve choice from create.get_object().name
• if choice not in c[“name”] for c in question.data.get(“oneOf”, question.any_of)
  • ignore
• else
  • retrieve answer key
    • if duplicate vote then ignore
    • else
      • update create object with question.id,
        • increment question replies
        • increment question_answers.answer_key by 1
      • also update remote id of create
        • poll_answer to question.id
        • answer choice to choice
        • stream to false?
        • poll_answer to true

handle replies (create)

• retrieve in reply to of create object
• if not found, ignore
• retrieve remote activity in reply to
• get reply from remote create object (ensure is create)
• if local reply to local question
  • i.e, starts with base url
  • is a question type
  • create is local reply
  • and create is not public
  • then handle question reply locally
  • else if private vote to remote question
    • set sent to true to reply .id in inbox

inbox

outbox

notifications

tasks

meta

notes

• tracks object id & separate remote_id in metadata
• box field represents inbox or outbox

Tasks

Setup main page with default output

Redirect requests to api to frontend

Work out how to define api

Implement actor endpoint

Look up structure of actor

See mastadon webpage

define datatype containing basic components

setup caqti with actor object

add function to construct actor from db

return actor on get of actor page

setup webfinger

see mastadon implementation page

End user notes

Running tests

• Update tests url with url and port of postgres
• Add test user to database with password password:

CREATE USER test PASSWORD 'password';

• Create test database:

CREATE DATABASE test;

Tasks

Setup formatting for register page

Work out how bulma setup on libre-ref

Setup styling

Get project building locally

Configure toast box on home page

Setup actor endpoint

Lookup definition of posts table

posts

id: int

userid: int

what: string

honker: string

xid: string

rid: string

dt: string

url: string

whofore: int

format: text

activities

id : int

data?

inserted at: UTC timestamp

updated at: UTC timestamp

actor: string

recipients: string list

apps

id

client name : string

bookmarks

id: int

userid: int

activity id: int

inserted at : int

updated at : int

deliveries

id: int

object id: int

user id: int

filters

id: int

user id: int;

filter id: int

phrase string

expires at time zone

whole word: boolean

inserted at: time zone

updated at: time zone

following

id: int

follower id: int

following id: int

state: int

inserted at: int

updated at: int

instances

id: int

host: string

unreachable since: timestamp

inserted at: timestamp

updated at: timestamp

favicon: string

notifications

id: int

user id: int

activity id: int

seen: boolean

inserted at: timestamp

updated at: timestamp

users

id: int

email: string

password: string

name: string

nickname: string

bio string

inserted at: timetstamp

updated at: timestamp

ap_id: string

avatar: jsonb

follower_address: string

follower_count

Update schema.sql to have post table

write function to validate signatures

open join mastadon blog post on followers

write basic function to validate functions

read micropub.blog encoding

Lookup how to do background tasks

update tests to use in memory database

Setup webfinger endpoint

look up web finger spec

look up how pleroma responds

Update user endpoint to return JSON response if application type is json, json+ld

Verify that actor returned has parity with pleroma

add inbox endpoint

print recived json, try signing, fail

add content length field to build signed headers enforce present

Refactor definitions into activitypub.common

Refactor decoders into activitypub.remote

update code to use database outside of build dir

validate follow request, return not acceptable if not valid

Add remote module to activitypub, parse pleroma follow request

Look at pleroma follow object

add follow form to page

write resolve function : username, domain -> remote user

given username, domain

lookup remote user -> if found, done.

if not found

create remote instance

send query to remote instance/.wellknown/webfinger?resource=acct:username@domain

if unreachable, record unreachable w. current time

if reachable + parse,

extract self + activity_json url

send request to url

if unreachable, record unreachable w. current timeif reachable + parse,add new remote userreturn created remote user

Create follow function : username, domain -> unit

resolve username, domain to remote user

write function to create follow request from current user, remote_user

{
	"@context": "https://www.w3.org/ns/activitystreams",
	"id": "https://my-example.com/my-first-follow",
	"type": "Follow",
	"actor": "https://my-example.com/actor",
	"object": "https://mastodon.social/users/Mastodon"
}

id -> Activity.fresh_id () |> Configuration.url

type -> follow

actor -> RemoteUser.url

object -> LocalUser.username |> Configuration.Url.format

create headers, content type activitystreams

uri is RemoteUser.url |> inbox

method is post

sign headers, with LocalUser.private key

print request body

add custom sql to retrieve all information needed to serialise pleroma follow object

Insert follow request into database

add local follow object to activitypub module

Use signing functions to manually send follow request to atestaccount

read joinmastadon blogpost for follow specification

Write frontend for following

on post, send follow to worker

parse username @ host

webfinger to remote to find remote user

construct follow request

post to server using signed

Update follow table to include (date created, date updated, & pending)

Update Resolver.follow to add pending entry to database

update home page to look good + post

Neaten up home pjage

Update home page get to resolve the users for each post as well

Fix logout button style

Complete Follow request implementation

add field to local user, manually accept follow requests

handle follows

Backend

on post to inbox with follow request

resolve local user

resolve remote user

create follow request from local to remote

check if local user account is manually accept follow requests

if not manual, send accept follow tasks to worker

define accept follow task in worker.mltakes a followcalculates inbox url of remote usercreate an accept follow request activity, add it to activities dbsend out follow request to inbox urlupdate status of follow request to be not pending

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://ocamlot.xyz/schemas/litepub-0.1.jsonld",
    {
      "@language": "und"
    }
  ],
  "actor": "https://ocamlot.xyz/users/example",
  "cc": [],
  "id": "https://ocamlot.xyz/activities/c99c2ba5-3ec0-412d-8cd0-2788867b7f67",
  "object": {
    "actor": "https://ocamlot.nfshost.com/users/example",
    "id": "https://ocamlot.nfshost.com/activity/7612618e-7c66-4792-b410-e4c971f5de51",
    "object": "https://ocamlot.xyz/users/example",
    "type": "Follow"
  },
  "to": [
    "https://ocamlot.nfshost.com/users/example"
  ],
  "type": "Accept"
}

on post to inbox with follow request accept

resolve local user

resolve remote user

lookup follow request in table, if not found ignore

update status to follow true

Frontend

add page to view followers

add page to view follow requests

on home page, below post, add button to view direct messages, follow requests

add follow request.ml to web to render follow requests

add function to view follow requests

on post of follow public id, send accept follow task to worker

Add ordered collection endpoint to users/username/followers

get to users/username/followers, if no offset, return ordered collection page

if ordered collection page

Update user activitypub to reference followers and following endpoint

Add button to profile tracking follow status of user, and on click, post follow form to server

add local follow task on post, with users username + local domain

add internal handling for local follows where domain matches config.domain

Debug why things break in pleroma

Update follow code to track whether there exists a follow with user already

Add partial index unique constraint

handle undo follow

try out unfollowing from ocamlot and see what happens

Write function to handle post of reject follow objects

first check reject.object.id is in activity database

then remove following request from table

(optional)? once posting added,

Handle posts

Update post to track all three of direct message, public, sensitive, followers only

on submit post on home page

collect all followers

on post to inbox with note

Add code to check activitypub origins (id of object must be of the same host as parent of object)

Handle likes

Handle boosts

Handle blocks

Handle reports

Handle attachments

Add field to config that points to image storage path

add image uploads database table

id

hash

name

modify user database schema with

is_admin

manually_accept_follow_requests

avatar image id

add profile page for local users - /users/example

if user = local user, enable edit button

add profile edit page

Update navbar of logged in with dms/home/local/twkn

Write function to handle post of accept follow objects

Write single SQL function to update entry in following database, using remote actor url + local user username

when decoding accept follow object in inbox

first, check accept.object.id. id is in activity database

lookup remote user by url (accept.actor)

retrieve url in to field, extract username from url

lookup local user by username

then update following relation between remote user, local user

Write function to handle post of note objects

update code to handle full tables, cc, to etc.

Tasks

setup home page

Go to html and see how home is structured

Use database to collect results (if user then users else public)

Display posts using result from database

Setup header

Handle logging in and out

Setup log in page

Pass in markdown about page in config

vendor markdown renderer

Create box with markdown details on login + register

setup webfinger

Implement posting

Update worker to only handle posts

Update post page to send request to worker

Update feed to display local feeds

refactor database to not use rec modules

Implement profile page at url //profile/username/

handle image uploading

Update actor object to track detailed stats

profile picture

summary

bio

display information

Implement faithful posting

Add field to form to describe public,followers,direct

update worker task type to track content type and public

update sql database to mention type of content

handle posting

Submission from user’s post is:

• title (optional)
• publicity of post (public, followers, direct)
• list of addresses to be sent to
• content type
• source

Upon receiving request:

1. for each target, attempt to resolve actor, ignore users who are not present
2. add post to database

Collect list of targets

split names in to_field, resolve

if post is public/followers, then collect all followers for user

for each target in parallel

Retrieve outbox for target

Post post object o target

Implement users page

List all public users who are local

List all known remote users who are public

add find for more users

Implement view post page

at url “//post/<id>/”

retrieve post by id

ensure that post is either public or user is follower/target of post

display post