Merge pull request #17061 from edx/ormsbee/pipeline_plans_doc

WIP stab at the plan for static assets in edx-platform

Merging this as an incremental step, even though there are still gaps.

docs/static_assets.rst

+#######################################
+edx-platform Static Asset Pipeline Plan
+#######################################
+
+Static asset handling in edx-platform has evolved in a messy way over the years.
+This has led to a lot of complexity and inconsistencies. This is a proposal for
+how we can move forward to a simpler system and more modern toolchain. Note that
+this is not a detailed guide for how to write React or Bootstrap code. This is
+instead going to talk about conventions for how we arrange, extract, and compile
+static assets.
+
+Big Open Questions (TODO)
+*************************
+
+This document is a work in progress, as the design for some of this is still in
+flux, particularly around extensibility.
+
+* Pluggable third party apps and Webpack packaging.
+* Keep the Django i18n mechanism?
+* Stance on HTTP/2 and bundling granularity.
+* Optimizing theme assets.
+* Tests
+
+Requirements
+************
+
+Any proposed solution must support:
+
+* Externally developed and installed Django apps.
+* Theming.
+* XBlock assets.
+* Existing tests.
+* Fast builds.
+* An incremental implementation path.
+* Other kinds of pluggability???
+
+Assumptions
+***********
+
+Some assumptions/opinions that this proposal is based on:
+
+* We want to shift as much as possible to Webpack and the JavaScript stack of
+  technologies, leaving the Python layer as thin as possible.
+* While we will try to make theming upgrades straightforward, we will be moving
+  around where files are located and where they're compiled out to.
+* We will be pushing towards a world that is more Django app-centric than LMS
+  vs. Studio centric, to reduce duplication.
+* At the same time, we want to consolidate assets far more efficiently than we
+  are doing today.
+* Leaning towards more static front ends + API calls.
+* However we still need to be compatible with Django's asset system for things
+  like third party apps (e.g. Django Rest Framework browsing assets, Swagger,
+  etc.)
+* It should be possible to pre-build static assets and deploy them onto S3 or
+  similar.
+
+Where We Are Today
+******************
+
+We have a static asset pipeline that is mostly driven by Django's built-in
+staticfiles finders and the collectstatic process. We use the popular
+``django-pipeline`` library, with UglifyJS as the JavaScript compressor (the
+binary is installed via node into node_modules). We also use the less well known
+``django-pipeline-forgiving`` extension to ``django-pipeline`` so we don't error
+out when files are missing (added when we started dynamically scanning XBlocks
+for assets).
+
+The ``django-pipeline`` config is aware of CSS files for the purposes of
+concatenation, but it does *not* know about the source Sass files (or handful of
+remaining CoffeeScript files). Those are processed with paver tasks before
+``django-pipeline`` ever sees them.
+
+We also have the following custom extensions to Django's builtin ``STATICFILES``
+mechanism:
+
+``openedx.core.djangoapps.theming.finders.ThemeFilesFinder``
+  Custom finder that overrides any static asset with a version from the themes
+  directory (``COMPREHENSIVE_THEME_DIRS`` defined in ``lms.envs.json`` and
+  ``cms.envs.json``).
+
+``openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder``
+  Custom finder that accesses and extracts assets from pip-installed XBlocks via
+  ``pkg_resources``.
+
+``openedx.core.storage.DevelopmentStorage/ProductionStorage``
+  Custom ``FileStorage`` classes that mostly exist for theme-awareness.
+
+LMS and Studio/CMS Separation
+-----------------------------
+
+LMS and Studio have their own directories for source assets (``lms/static`` and
+``cms/static``), and have symlinks to shared assets in ``common/static``. We
+treat the static asset compilation and collection phase for LMS and Studio as
+separate projects that happen to share a lot of pieces. They output to different
+places (typically ``/edx/var/edxapp/staticfiles`` for LMS and
+``/edx/var/edxapp/staticfiles/studio`` for Studio) and can be collected
+separately. However in practice they're always run together because we deploy
+them from the same commits and to the same servers.
+ 
+Django vs. Webpack Conventions
+******************************
+
+The Django convention for having an app with bundled assets is to namespace them
+locally with the app name so that they get their own directories when they are
+gathered together into a common static directory by collectstatic. For example,
+the edx-enterprise app has a ``static/enterprise`` folder, so its assets are
+compiled to ``/edx/var/edxapp/staticfiles/enterprise`` by edx-platform and will
+not conflict with assets from any other Django app.
+
+Webpack conventions would have us create a single set of configuration files at
+the root of edx-platform, which would specify all bundles in the project.
+
+TODO: The big, "pluggable Webpack components" question.
+
+Proposed Repo Structure
+***********************
+
+All assets that are in common spaces like ``common/static``, ``lms/static``,
+and ``cms/static`` would be moved to be under the Django apps that they are a
+part of and follow the Django naming convention (e.g.
+``openedx/features/course_bookmarks/static/course_bookmarks``). An app's
+``templates/{appname}`` directory will only be for server side templates, and
+any client-side templates will be put in ``static/{appname}/templates``.
+
+Proposed Compiled Structure
+***************************
+
+This is meant to be a sample of the different types of things we'd have, not a
+full list:
+
+::
+
+  # Webpack bundles/post-processed assets
+  /webpack/css
+          /fonts
+          /js
+          /vendor ?
+
+  # Django apps that are in the edx-platform repo
+  /course_bookmarks
+  /course_experience
+
+  # edX authored, installed via separate repo
+  /enterprise
+
+  # Entirely third party apps that we need to maintain compatiblity with.
+  /admin
+  /rest_framework
+
+  # Themes are part of the "theming" app
+  /theming/themes/open-edx
+                 /red-theme
+                 /edx.org
+  
+  # XBlocks still collect their assets into a common space (/xmodule goes away)
+  # We consider this to be the XBlock Runtime's app, and it collects static
+  # assets from installed XBlocks.
+  /xblock
+
+Django vs. Webpack Roles
+************************
+
+Rule of thumb: Django/Python still serves static assets, Webpack processes and
+optimizes them.
+
+Webpack would be responsible for all Sass compilation in edx-platform. It would
+also be responsible for the optimization/minification of JavaScript assets, but
+those optimized assets would only appear under the ``/webpack`` directory. Third
+party assets that Webpack is not aware of may have hash suffixes applied to them
+by the Django collectstatic layer, but will not otherwise be processed or
+optimized in any way -- so no coffeescript/sass compilation, no uglifyjs
+minification, etc.
+
+The django-pipeline dependency should be removed altogether.
+
+Themes
+------
+
+Theme handling is muddled. The fact that themes can override server-side
+templates means that Python has to be aware of them. At the same time, we want
+to shift over Sass compilation as a whole to Webpack, meaning that at least some
+knowledge about where they are and how to compile them has to exist there. Also,
+there are JS assets in some themes that provide additional functionality, and it
+would be a performance degradation if those assets were no longer optimized.
+
+What I do NOT want to happen:
+
+* Significant end user performance degradation.
+* Having an *additional* system in the asset pipeline (e.g. keeping
+  django-pipeline around while having additional systems).
+
+I think that means that conceptually, there exists a larger Static Asset system
+that exists and that we think of both Webpack and Django being consumers of its
+configuration. This is also very fuzzy at the moment.
+
+Asset Groups
+------------
+
+There will be logical groupings of static assets. There should be uniformity and
+no duplication within a group, but we would allow duplication between groups to
+better facilitate independent deployment and isolation.
+
+Example Groups:
+
+* XBlock/XModule Assets
+* LMS/Studio apps in edx-platform
+* Third party app, such as edx-enterprise