Expand drawing tools i highlight mode

[Cimbali]

From @ceermonk in #66

Suggestion for some future release: It would be great to have few drawing tools in the highlight mode,
e.g., straight line, rectangles, oval, text -- something along the lines of Xournal or MS Journal.

I'm not sure how reasonable this is though, but let's create the feature request and see if people like it.

[raphaelquast]

I'd love it!
something similar to the features of gInk for example would be awesome!!
https://github.com/geovens/gInk

[Cimbali]

I think everything gIink does we can already do, that’s more of a question of interface. If you select a sufficient stroke width and some transparency in your color.

So we maybe should make some better shortcuts for different presets ?

[raphaelquast]

Since I have just recently tested this... here's why I've been using gInk instead of your feature:
(... not to mention that the rest of your software is already awesome!)

• you do not yet have the possibility to use an eraser (I do not mean the undo button.. i mean a proper eraser)

• i agree, aside of the eraser, shortcuts are the key here.. from my experience, what I would need is:

  • numbers 1-9 to change predefined pens (+ a table somewhere on the screen that helps me remember the numbers)
  • a shortcut to switch to the eraser
  • undo-redo shortcuts
  • a "show/hide" annotations shortcut

... and there's one more thing which might be a bit more complicated to look into...

• I'm using a writepad + stylus which is (in pen-mode) not working properly with you annotate feature
  (there's a quite long lag that results in a straight line at the start of each line i want to draw)
  • it works if i use the stylus like a mouse, but then the writing experience is rather limited...
    (i can try to test this more properly if you want to look into it)

If this would be available, your feature would be much better since it allows to draw directly on the
presenter-screen rather than on the main-screen!

[Cimbali]

Alright, so the ideas here are:

• eraser
• predefined pens
• shortcuts
  • to switch between pens, eraser, etc. (maybe pens 1-9, eraser 0?)
  • to toggle annotations (toggling annotation mode is already H)
  • undo/redo (isn’t ctrl-z or U already undoing?)
• drawing too laggy (will have to reproduce that, I don’t have a writepad or stylus)

From @MatanZ’s PRs we see there is some interest in toggling how persistent the annotations are, maybe we a toggle in the annotation mode is useful ?

[Cimbali]

If I do a real ugly mockup:

So we could add a bottom dock similar to gInk (I’ve just copied theirs for this mockup but we would make our own)
I think we could add a selector that says something like

• Erase drawings on page change
• Remember per-page drawings (save / restore)
• Do not erase (document-wide drawings)

Maybe the pen editor could go closer to the pen selection dock etc.

[MatanZ]

Pen edit: right click on pen to copy current settings.

I was thinking of two columns of small squares under the pen width dial. There is some unused screen there.

About performance - I have a tablet. a scribble has thousands of points, and there is no latency in writing/erasing.
My CPU is an i5-3550. I don't know if people run pympress on much slower systems (raspberry pi?)

In my branch I have writing tablet support (separate from X11 presenting tablet as a mouse). I will have predefined pens (selected by number keys) soon.

I am not sure of the need for eraser mode, erasing by right button (or stylus button) works well enough for me.

Two other tools I plan to add are selection (by touching/marking a rectangle) in order to allow changing color/width and maybe moving, and a filled rectangle tool.

[Cimbali]

Doing it vertically is fine too, but the buttons are intentionally big to allow for touch screen usability (say if you run pympress on a tablet or some other surface-like touch screen used without a stylus).

[Cimbali]

Here’s an implementation of different presets. Currently looking like this:

Maybe the presets should be a horizontal bar, but we need to also ensure pympress still works on narrow screens (see also #154).

The eraser works like an eraser on any paint-like program, rather than deleting lines.

Presets are saved in the config file, preset 9 is inherited from the current single-color setting if it exists in the config file before this version.

[Cimbali]

I think possibly the color and width selector could go along the presets, and the other column can contain a scribble-persistence toggle (eraser on page change, keep for whole document, maybe also save/restore per page)

[MatanZ]

I think the layout needs to be more flexible, so as to make the page as large as possible.

• If the aspect ratio is similar to that of the pane, have one vertical and one horizontal bars.
• If the aspect of the page is more portrait than that of the pane, have two vertical bars (preferably on two sides of the page, so that it remains in the same place with and without the toolbars, see my branch for example).
• If the aspect of the page is more landscape than the pane, two horizontal bars.

[raphaelquast]

if you can provide me with a windows binary that includes the new changes I'd be happy to provide some feedback on the usability of the new interface! ... after all I'm very much looking forward to this 😄
(I tried to build pympress from source on windows... but i did not succeed...)

[Cimbali]

I can’t guarantee the build will succeed on first try, but I launched it anyway:
https://ci.appveyor.com/project/Cimbali/pympress-msys2/build/job/9r1cxht38nj1c34g

[raphaelquast]

thanks for the try! (however... it seems that the build stopped with "execution time exceeded maximum allowed time")

[Cimbali]

I tried to figure out what happens but couldn’t, because I don’t have a lot of time. It seems the build runs fine, the tests run and succeed, and then the whole process hangs for an unknown reason…

[raphaelquast]

OK, thanks anyways! I'll simply wait for a release that includes the changes!

[Cimbali]

I removed all the tests and the zip is suspiciously small but you can give the msi a try @raphaelquast:

https://ci.appveyor.com/project/Cimbali/pympress-msys2/builds/36138307/job/banrfi7gupbelain/artifacts

I haven’t even downloaded it myself so no guarantees.
I just tested it and it seems to work, so I’m waiting for your feedback on the UI and pre-selected colors :)

[raphaelquast]

awesome, I tried and the .msi works perfectly! thanks!
since you've spent some time on this... I thought i'll do the same 😄

first of all, the presets are very nice and do their job perfectly!
(maybe blue and green are a bit dark, but this certainly depends on the used screen
and one can anyways set it's preferred colors in a second...)

... however, there are still some issues which I've tried to collect in the following:

general:

• a eraser-hotkey 0 is not very practical... -> I'd suggest holding crtl to temporarily select the eraser so that you can quickly erase something and directly continue writing
• highlite features do not remain visible on slide-change (currently you have to press H again on each slide in case you want to continue writing on the slides) -> I'd suggest keeping the highlighte-mode active until H is pressed again

... and some additional suggestions (might need some more work)

• how about making the width of the eraser adjustable?
• is it possible to show a dot with the size of the stroke-width instead of a mouse-indicator?
  • maybe even add a button to show/hide this pointer to the audience?

performance

(I have a laptop with 16Gb ram and a Intel i7-7500 CPU so this should not be the issue...)

• the lines (especially thick ones) show nasty artefacts on fast direction-changes
• the lines appear rather edgy (e.g. like a collection of straight line-segments)
  ... this can be problematic if one wants to actually write text or equations (and not just draw rectangles)
• the eraser is affected by the size of the window

to make this more clear I made some gifs:

this is what i get if i use pympress with the writepad and a stylus

• you can also see the issue with the writepad (it always takes a short moment till the drawing starts which results in a straight line at the beginning of each stroke... and it is impossible to actually write letters or numbers properly)

this is what i get if i use pympress with the writepad in mouse-mode

... and for comparison,

this is how gink looks like...

... it seems that the eraser is affected by the size of the window!

[Cimbali]

Thanks for the feedback ! I also think the green and blue could be somewhat lighter.

• We could draw the current mouse position on the presenter side only, not on the content slide. And draw the eraser as an empty circle would be useful too.

• Removing the highlight-mode from one slide to the next is a feature not a bug IIRC, but we could make that configurable. This should be take into account with the drawing persistence.

• On the drawing smoothness/responsiveness I don’t think we can control how often the events (i.e. updates of the mouse position) are sent.

  • We’re potentially slowed down by the code being python and not something low-level such as C ? gInk is in C# Then we might miss mouse events while we’re drawing. Not entirely sure of this, the events might be queued too.
  • Not sure I can do much about your stylus issue. (The initial waiting time one.) Since your notepad works fine it seems it is more of a stylus than a pympress problem. It also seems, on the gif, as though there is some animation to make you wait some time before moving ? Something to distinguish a stylus click from a drag/short flick ? Is it maybe a setting of your stylus ? If so. could you reduce or remove it ?

  All this means that if we want to make the lines smoother we should probably use bezier splines instead of just line segments. However I’m not sure how to go about that: either every other mouse position is a control point instead of an actual point, or we need to somehow define control points from the mouse movement, maybe using the speed of the movement or something like that.

• Drawing positions (pen or eraser) are relative to the slide, however the line thickness is not. If we make them relative we’ll fix the resize issue. (We must be careful to make them relative to something that makes sense, as we don’t want people to have to resize their pens when they change pages to a page with a different aspect ratio for example.)

• The eraser is currently bigger than the selecting scale allows (50 points and the scale is 1..30), hence it’s size is not settable. That too can be easily fixed, by changing the range of the scale when the eraser is selected.

• I’m not sure a control button eraser is the best ? Especially for size setting. If you’re in stylus mode it’s probably easier to click the buttons anyways rather than go to the keyboard.

[MatanZ]

• I do not believe mouse events are missed. The system has an event queue that keeps events until the application is ready for them.
• I also don't believe there should be a problem with performance. On my system with a similarly powered CPU, pympress has no problem with curves that have thousands of points, and are thus very smooth. I do read the pen directly from /dev/event, rather than through X11.

[raphaelquast]

no problem! since I really like what you are doing here I am happy to bring in my 2 cents 😄

• concerning the eraser-button... my comment was not so much about if you use crtl or another button, what I meant is that a button that activates the eraser only as long as it is pressed would be more helpful than a button that switches permanently to the eraser (...in this way you don't need to remember what pen you have selected prior to switching to the eraser which makes the workflow during a presentation much smoother)

• i agree that an option to switch between a persistant and non-persistant highlite-feature would be a very useful feature
  (there are certainly applications for both settings)

• concerning the performance-issues... I don't know what's going on behind the scenes but I also don't think that the usage of python would result in a loss of such a huge amount of mouse events

  • @MatanZ does your comment mean you do not see the same behaviour on your pc?
    (maybe then it's a windows-related problem?)

  • ... and yes, I would definitely go for smoother lines... those corner-artefacts don't look particularly nice and intensify with line-strength... However, lines should also not be too smooth since this would interfere with writing...

    • not sure about this, but maybe something like a "Cardinal Spline" could work in a more straight-forward way?

• concerning the stylus-issue... you're right, the pen seems to initialize a touch-event which causes the lag... I'll have a closer look at what's happening here on my side...

[Cimbali]

The thing is quadratic bezier splines are easy to do with cairo (there’s a context.curve_to() function that does them) so anything that’s using that instead of re-writing a spline interpolation from scratch is going to be the easiest.

I think what I did in the latest version isn’t quite perfect but already a step in the right direction.

Now:

• lines are smoothed, though still not as smooth as gink
• eraser (and generally line width) scale with window size
• there is a pen-preview replacing the mouse
• I feel like it gets laggy if you really draw a lot. Maybe I could pre-compute the spline parameters rather than do it on the fly, but it’s likely it’s the actual interpolating that takes some time.

Either one of these builds should work for windows:

• https://ci.appveyor.com/project/Cimbali/pympress/build/job/p0pfodbjgr3hfaa2
• https://ci.appveyor.com/project/Cimbali/pympress-msys2/builds/36163014/job/3471rwqj0v635shd

[Cimbali]

So there is a definite increase in time between events when there are a lot of curves to draw, in this latest version.

[MatanZ]

* Drawing positions (pen or eraser) are relative to the slide, however the line thickness is not. If we make them relative we’ll fix the resize issue. (We must be careful to make them relative to something that makes sense, as we don’t want people to have to resize their pens when they change pages to a page with a different aspect ratio for example.)

This commit MatanZ@bcf956e changes the pen width to points, instead of pixels. It will not apply cleanly to your tree, but it is trivial enough.

[Cimbali]

I have a little less time right now, but managed to do some caching of scribbles to reduce lag. I don’t know if other things could be worth adding to this interface ?

• There’s a risk of making it too heavy of course, but I ran into zoom’s annotation toolbox the other day and quite liked the « stamp » option, to place e.g. checkmarks on the screen. I don’t remember other tools specifically so they’re probably less interesting.

  After looking into it the extra things are:

  • stamp (and arrow spotlight): arrow, checkmark, cross, star, heart, question mark.
    All those could be simply utf-8 symbols: ❌ ✔ ⇨ ⭐ ❤️ ❔ (or ❓, or simply ?)
  • text. Risks clashing with the current annotation system. Also linked to Edit PDF annotations #86.
  • selection + moving/scaling/etc the drawn objects
  • save/export the drawn objects
  • drawing of shapes (arrows, rectangle shapes, ellipse shapes, filled rectangles, filled ellipses)

  What we already have:

  • pointer spotlight
  • freehand drawing
  • erasing
  • undo/redo/clear

  I’m not sure we need shapes or selection. Stamps seem nice, supporting text probably means we need to look into what kind of annotations should be on-screen text and which should be displayed as annotations (which is the default currently).

• The persistence of annotations must still be tackled, I’ll probably reuse some of @MatanZ’s PRs here (still have to look into detail into what’s done on this topic).

• Another thing we can look into, is to save/load annotations from PDF annotations. For example, current annotations could be saved as Poppler.Annot.POLY_LINE and stamps as Poppler.Annot.STAMP − but this would definitely be a topic for a different issue.