Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
E
eScriptorium
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 78
    • Issues 78
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 4
    • Merge Requests 4
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
  • scripta
  • eScriptorium
  • Issues
  • #231

Closed
Open
Opened May 06, 2020 by Robin Tissot@rtissotMaintainer

Tasks queue improvements

Requirements:

  • some tasks need to be chained (eg image compression -> thumbnail generation)
  • some tasks may need to be done as fast as possible (eg single line polygon recalculation which is done live for now)
  • the user should be able to see the number of tasks still in queue (and maybe a warning if the server is currently on heavy workload)
  • identify tasks that could be done easily on another server, to unburden the database/gpu server, for scaling
  • load balancing for cpu, gpu and memory, one user shouldn't use all resources if other users are queuing tasks
  • keep enough resources for db and web server
  • better feedback on tasks that did not reached the worker yet
  • when deleting a document or parts, cancel related tasks/processes.

Technical considerations:

  • check other frameworks than celery, mostly to see if load balancing is easier to do
  • could spawn a dedicated queue by user with --autoscale to simplify dividing workload to all concurrent users?
Edited Jul 09, 2020 by Robin Tissot
To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: scripta/escriptorium#231