Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve handling of full items #297

Closed
wants to merge 2 commits into from
Closed

Improve handling of full items #297

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
18c8a0f
Show warning message for #290
AGSPhoenix Apr 20, 2019
190cc18
status check: check if item is full
AGSPhoenix Apr 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion internetarchive/cli/ia_upload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,13 @@

import six
from docopt import docopt, printable_usage
from requests.exceptions import HTTPError
from requests.exceptions import HTTPError, ConnectionError
from schema import Schema, Use, Or, And, SchemaError

from internetarchive.cli.argparser import get_args_dict, convert_str_list_to_unicode
from internetarchive.session import ArchiveSession
from internetarchive.utils import validate_ia_identifier, get_s3_xml_text
from internetarchive import get_item

# Only import backports.csv for Python2 (in support of FreeBSD port).
PY2 = sys.version_info[0] == 2
Expand All @@ -87,6 +88,10 @@ def _upload_files(item, files, upload_kwargs, prev_identifier=None, archive_sess
responses += response
except HTTPError as exc:
responses += [exc.response]
except ConnectionError as exc:
print("Received ConnectionError while uploading. If this happens consistently,",
" the item you're uploading to might be full. Run --status-check!")
raise
finally:
# Debug mode.
if upload_kwargs['debug']:
Expand Down Expand Up @@ -176,6 +181,10 @@ def main(argv, session):
'Expect 503 SlowDown errors.'.format(args['<identifier>']),
file=sys.stderr)
sys.exit(1)
elif get_item('{0}'.format(args['<identifier>'])).item_size >= 1099511627776:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we use a global, in capitals, at the top of the file here instead of this magic number? something like:

IA_MAX_SIZE = 1099511627776

or even better, what the actual value means:

# one tibibyte
IA_MAX_SIZE = 2^40

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @anarcat. Also, we already initialize the item on https://github.com/jjjake/internetarchive/blob/master/internetarchive/cli/ia_upload.py#L183, this check should happen after that so we only have to do it once.

print('warning: {0} is full and cannot accept uploads.'.format(args['<identifier>']),
file=sys.stderr)
sys.exit(1)
else:
print('success: {0} is accepting requests.'.format(args['<identifier>']))
sys.exit()
Expand Down