Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Imgee V2 #59

Merged
merged 43 commits into from
Jul 17, 2017
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
441629f
removed celery as dependency
Apr 7, 2017
7337d16
fixed - was missing folder name when downloading file
Apr 10, 2017
ba53794
handling multiple resize request for same file
Apr 11, 2017
412e44b
removed print statements
Apr 11, 2017
9a49863
if there is an error while processing a file, raise it
Apr 11, 2017
127da69
fixed comment
Apr 11, 2017
be447ac
removed all tests
Apr 11, 2017
3b54fbd
removed more unnecessary things
Apr 11, 2017
6cd46ce
removed unnecessary imports and minor cleanup
Apr 11, 2017
57a99f0
renamed async.py
Apr 11, 2017
643e476
Merge pull request #56 from hasgeek/celery-removal
shreyas-satish Apr 12, 2017
9ab5a45
added label tests
Apr 14, 2017
d57fec0
separating label logic
Apr 14, 2017
e0f547f
fixed label tests
Apr 14, 2017
cb7b9bd
more cleanup, moved some views out of index.py
Apr 17, 2017
c6c4645
cleaned up label saving codes a little
Apr 17, 2017
b3ff9f4
added tests for upload and delete
Apr 17, 2017
c663086
updated status code for delete test
Apr 17, 2017
e4ae277
mixed couple of migrations and showing error
Apr 17, 2017
eb5cab0
Merge pull request #57 from hasgeek/new-tests
shreyas-satish Apr 17, 2017
b4a14c1
some more cleanups
Apr 18, 2017
6fabf71
Merge pull request #58 from hasgeek/new-tests
shreyas-satish Apr 18, 2017
fc28fea
using get_image_url to create image url
Apr 18, 2017
edc1bf9
not showing delete label button if not logged in
Apr 18, 2017
95c1615
importing only the needed forms
Apr 18, 2017
95adecf
added tests for resizing files and testing with multiple file types
Apr 20, 2017
4451fb0
comparing redirect location with generated thumbnail url
Apr 20, 2017
4635676
added redis to travis
Apr 20, 2017
838d1ec
fixed conflicts
May 4, 2017
418e080
Merge branch 'master' of github.com:hasgeek/imgee into imgee-v2
May 9, 2017
f112e6c
fixed fixtures file to use new init_app behavior
May 9, 2017
cc369e6
fixed conflict
Jun 1, 2017
380bb87
Function name fix, using uuid1mc, time format fix, TaskRegistry fix (…
Jun 23, 2017
f2c6fd6
fixed conflict
Jun 23, 2017
23007ec
lot of minor fixes
Jun 29, 2017
59bb581
using dotted relative import syntax everywhere, added tests for regis…
Jun 30, 2017
d31b027
cleaning up queries and imports
Jul 5, 2017
ff390de
cleanup of os.path and more comments on thumbnail
Jul 13, 2017
6731ef9
fixed typo
Jul 13, 2017
3dcb9e5
added buildspec.yml file for testing AWS CodeBuild
Jul 13, 2017
d2c1228
setting env varialbe manually for aws codebuild
Jul 13, 2017
88457f5
removed buildspec.yml file
Jul 17, 2017
102e603
showing loading spinner if file is still processing
Jul 17, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,7 @@ baseframe-packed.css
baseframe-packed.js
error.log
imgee/static/uploads
*.bak
instance/production.env.sh
imgee/static/gen
tests/imgee/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What goes in this folder?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uploads made during running the tests.

-> % tree tests/imgee 
tests/imgee
└── static
    └── test_uploads
        ├── 0a795f311f90494292df03f64317c57d.svg
        ├── 1ce47cefeec241b9a7266ed852131e48.svg
        ├── 28029c34ba9947ad8e6ce53e074b3617.gif
        ├── 318c1dbbefa7451e82fee1d7c77e3282.svg
        ├── 31b9a8a9be6947db9913fdb38c0fd4e2.jpeg

Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,14 @@
def upgrade():
op.add_column('thumbnail', sa.Column('width', sa.Integer(), nullable=True))
op.add_column('thumbnail', sa.Column('height', sa.Integer(), nullable=True))

connection = op.get_bind()
tn = Thumbnail.__table__
result = connection.execute(select([column('id'), column('size')], from_obj=tn))
w_h = [dict(tnid=r.id, w=int(r.size.split('x')[0]), h=int(r.size.split('x')[1])) for r in result]
updt_stmt = tn.update().where(tn.c.id == bindparam('tnid')).values(width=bindparam('w'), height=bindparam('h'))
connection.execute(updt_stmt, w_h)
if len(w_h) > 0:
updt_stmt = tn.update().where(tn.c.id == bindparam('tnid')).values(width=bindparam('w'), height=bindparam('h'))
connection.execute(updt_stmt, w_h)


def downgrade():
Expand Down
5 changes: 3 additions & 2 deletions alembic/versions/2c7e25599132_add_mimetype.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,9 @@ def upgrade():
sf = StoredFile.__table__
result = connection.execute(select([sf.c.id, sf.c.title]))
mimetypes = [dict(sfid=r[0], mimetype=guess_type(r[1])[0]) for r in result]
updt_stmt = sf.update().where(sf.c.id == bindparam('sfid')).values(mimetype=bindparam('mimetype'))
connection.execute(updt_stmt, mimetypes)
if len(mimetypes) > 0:
updt_stmt = sf.update().where(sf.c.id == bindparam('sfid')).values(mimetype=bindparam('mimetype'))
connection.execute(updt_stmt, mimetypes)


def downgrade():
Expand Down
3 changes: 1 addition & 2 deletions alembic/versions/2d5db2a698f6_init.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@ def upgrade():
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('updated_at', sa.DateTime(), nullable=False),
sa.Column('userid', sa.Unicode(length=22), nullable=False),
sa.Column('description', sa.UnicodeText(), nullable=False),
sa.Column('type', sa.Integer(), nullable=False),
sa.Column('name', sa.Unicode(length=250), nullable=False),
sa.Column('title', sa.Unicode(length=250), nullable=False),
sa.PrimaryKeyConstraint('id'),
Expand All @@ -63,6 +61,7 @@ def upgrade():
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('updated_at', sa.DateTime(), nullable=False),
sa.Column('name', sa.String(length=50), nullable=False),
sa.Column('title', sa.String(length=50), nullable=False),
sa.Column('profile_id', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['profile_id'], ['profile.id'], ),
sa.PrimaryKeyConstraint('id')
Expand Down
11 changes: 9 additions & 2 deletions alembic/versions/347ba3ac054f_update_image_attrs.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@
import os.path
import sys
from glob import glob
from alembic import op
from sqlalchemy.sql import select
from sqlalchemy.orm.session import sessionmaker
from sqlalchemy.orm import load_only

sys.path.append('../../')
from imgee import db
Expand All @@ -22,7 +26,10 @@


def upgrade():
imgs = StoredFile.query.filter_by(size=None)
connection = op.get_bind()
Session = sessionmaker(bind=connection.engine)
session = Session(bind=connection)
imgs = session.query(StoredFile).filter_by(size=None).options(load_only("id", "name", "title"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe Alembic gives you a session already. Is a new session necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jace can do that with op.execute() but then have to define the schema for stored file in this migration file itself. I wanted to avoid it. so creating a session and using that so that I can use the ORM.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a problem. Migrations must never import models. They must always be locally defined in the migration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is that the model may no longer exist in the same form in the code when the migration is run. You need version controlling.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jace that's exactly why I did this. The way it was before, the model was imported and used which was breaking the migrations. The way above, it uses the session of the migration, so it uses the version of the model at that point in migration and not the current version of the model.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't make sense. You're importing from models. You always get the current version of the model.


for img in imgs:
path = path_for(img.name) + '.*'
Expand All @@ -34,7 +41,7 @@ def upgrade():
print 'updated attributes of %s\n' % img.title,
else:
print 'local file not found for %s\n' % img.title,
db.session.commit()
session.commit()


def downgrade():
Expand Down
13 changes: 4 additions & 9 deletions imgee/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
# The imports in this file are order-sensitive

import os
from celery import Celery

from flask import Flask, redirect, url_for
from flask.ext.lastuser import Lastuser
Expand All @@ -12,20 +11,18 @@
import coaster.app
from ._version import __version__


version = Version(__version__)
app = Flask(__name__, instance_relative_config=True)
lastuser = Lastuser()
celery = Celery()

assets['imgee.css'][version] = 'css/app.css'

from . import models, views
from .models import db
from .api import api
from .async import TaskRegistry
from .tasks import TaskRegistry

registry = TaskRegistry(os.getenv('ENV', 'production'))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any dependency on system environment should be in coaster.app. The system environment is not a secure place for keeping settings.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I set this inside the init_for(env) function and then use env? TaskRegistry has name='default' set by default. I wanted to use something more verbose for the redis keys. hence did this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcode name='imgee'?


registry = TaskRegistry()

def mkdir_p(dirname):
if not os.path.exists(dirname):
Expand All @@ -40,14 +37,12 @@ def error403(error):
def init_for(env):
coaster.app.init_app(app, env)
baseframe.init_app(app, requires=['baseframe', 'picturefill', 'imgee'])
app.error_handlers[403] = error403
app.error_handlers[403] = error403
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this Flask's recommended way of defining an error handler? If not, we should use the recommended way.

lastuser.init_app(app)
lastuser.init_usermanager(UserManager(db, models.User))
if app.config.get('MEDIA_DOMAIN') and (
app.config['MEDIA_DOMAIN'].startswith('http:') or
app.config['MEDIA_DOMAIN'].startswith('https:')):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs, startswith takes a tuple. You can use: app.config['MEDIA_DOMAIN'].startswith(('http:', 'https:'))

app.config['MEDIA_DOMAIN'] = app.config['MEDIA_DOMAIN'].split(':', 1)[1]
mkdir_p(app.config['UPLOADED_FILES_DEST'])
celery.conf.add_defaults(app.config)
registry.set_connection()
app.register_blueprint(api, url_prefix='/api/1')
60 changes: 0 additions & 60 deletions imgee/api.py

This file was deleted.

107 changes: 0 additions & 107 deletions imgee/async.py

This file was deleted.

12 changes: 6 additions & 6 deletions imgee/forms.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,17 @@
# -*- coding: utf-8 -*-

import os.path
from coaster import make_name
from flask.ext.wtf import Form
from baseframe.forms import Form
from wtforms.validators import Required, ValidationError, Length
from wtforms import (FileField, TextField, HiddenField,
SelectMultipleField, SelectField)
from wtforms import (FileField, TextField, HiddenField, SelectField)

from imgee import app
from imgee.models import Label
from imgee.utils import get_file_type, is_file_allowed
from imgee.utils import is_file_allowed


def valid_file(form, field):
if not is_file_allowed(field.data.stream):
if not is_file_allowed(field.data.stream, field.data.mimetype, field.data.filename):
raise ValidationError("Sorry, unknown image format. Please try uploading another file.")


Expand All @@ -28,6 +26,7 @@ class DeleteImageForm(Form):
class PurgeCacheForm(Form):
pass


def reserved_words():
"""get all words which can't be used as labels"""
words = []
Expand Down Expand Up @@ -67,6 +66,7 @@ class EditTitleForm(Form):
file_name = HiddenField('file_name')
file_title = TextField('title', validators=[Required(), Length(max=250)])


class UpdateTitle(Form):
title = TextField('Title', validators=[Required(), Length(max=250)])

Expand Down
32 changes: 27 additions & 5 deletions imgee/models/stored_file.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-

from coaster.sqlalchemy import BaseNameMixin, BaseScopedNameMixin
import imgee
from imgee import app, url_for
from imgee.models import db
from imgee.utils import newid, guess_extension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use dotted syntax as far as possible:

from .. import app, url_for
from . import db
from ..utils import newid, guess_extension

Also, what is this url_for? The same that Flask provides?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any advantage of using the relative import over absolute one? I usually feel a little confused whenever I face relative import. If the module name is present instead, it's clear on first glance where it's being imported from.

And yeah, that's the same from flask.


Expand Down Expand Up @@ -58,7 +58,29 @@ def __repr__(self):
def extn(self):
return guess_extension(self.mimetype, self.orig_extn) or ''

def is_queued_for_deletion(self):
if imgee.app.config.get('CELERY_ALWAYS_EAGER'):
return False
return imgee.registry.is_queued_for_deletion(self.name+self.extn)
def dict_data(self):
return dict(
title=self.title,
uploaded=self.created_at.strftime('%B %d, %Y'),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use .isoformat() + 'Z'

filesize=app.jinja_env.filters['filesizeformat'](self.size),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just call this function directly? Unless it's an inbuilt Jinja filter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an inbuilt filter of jinja.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it needs a binary parameter too. https://github.com/pallets/jinja/blob/master/jinja2/filters.py#L459

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the binary format? By default it's False.

imgsize='%s x %s' % (self.width, self.height),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

u'%s×%s px'

url=url_for('view_image', profile=self.profile.name, image=self.name),
thumb_url=url_for('get_image', image=self.name, size=app.config.get('THUMBNAIL_SIZE'))
)

def add_labels(self, labels):
new_labels = set(labels)
old_labels = set(self.labels)
if new_labels != old_labels:
self.labels = labels

if (new_labels == old_labels):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brackets aren't really necessary here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed them.

status, diff = '0', []
elif (new_labels > old_labels):
status, diff = '+', (new_labels - old_labels)
elif (old_labels > new_labels):
status, diff = '-', (old_labels - new_labels)
else:
status, diff = '', new_labels

return status, list(diff)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if you had both additions and deletions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part of the code is a bit hairy. I think it can be handled better. I'll try something and show you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change here?

Loading