Lecture: Handling Static Files

Handling Static Files

Introduction

Even though Django is really good for processing complex files and weaving together data from disparate sources, it is unnecessary for django to serve static files that don't change. This consumes more resources and makes it difficult to utilize content-delivery networks (CDNs) for more efficiently delivering static content. Furthermore, unless you program in a custom path for non-html files, Django doesn't have the built-in ability to serve non-html files anyways. Thus, for production environments, Django gives you the ability to split your content between dynamically-produced html content served by Django's server, and static content that is saved as a file and served by a separate server.

This is accomplished by modifying the URLs pointing to static files. Django allows you to specify static files using a special {% static ... %} tag in a HTML template in order to specify different paths for static vs dynamic content. Using the static tag forces Django to rewrite the URLs for static files, pointing static files to a different host url.

One of the hurdles you will face is that many of the static URLs you will be using in your website aren't just embedded in the HTML templates or structure of your Django site, but are part of your markdown files -- the posts that you will be converting into HTML and rendering via Django templates. Furthermore, these files may be referenced by an absolute path -- somewhere specific on your hard drive -- as well as by a relative link.

However, in this context, even if you modify the URLs in your markdown files to include the custom Django static templating tag, your markdown files are pre-rendered as a html string by the python-markdown library -- as it is currently, it isn't processed by Django's templating engine in order to convert the static tag the way it would otherwise be.

So, in order to implement static files for your website, you to consider how to modify your markdown processing chain to support static file routing, and how you layout your markdown / media file tree.

The mechanics of static tags

How do you specify a link to a static file? Typically you replace the URL. This is made possible via a Django-specific templating tag called static, changing an ordinary anchor like this:

<img src="/path/to/file">

to this:

<img src="{% static 'path/to/file'%}">

However, we are working with markdown. We enter links to images like this:

![text](/path/to/file)

So why would it be a problem for us to simply do the following??

to this:

![text]({% static 'path/to/file'%})

First of all, markdown would have no understanding of file links that look like that. The Markdown processor would probably not accept that as a valid path This is because those tags are processed exclusively by Django. Second, even if a markdown renderer was able to process that correctly, Django doesn't interpret rendered HTML supplied inside its template the same way it renders its own template. It doesn't recognize tags embedded within other tags. Finally, even if It were technically feasible. It wouldn't want to change thousands of image links in your source.

So for those reasons, just replacing links in your markdown isn't enough. What we have to do is modify the markdown rendering process to capture paths to local files before inserting the final html into a django template.

Note: somewhere above your first use of the static tag you have to load the static tag like this:

{% load static %}

This just needs to be inserted once somewhere above in the same file, so the Django processor knows to load the static tag.

The solution? Python-markdown extensions

The best way to find and replace files, therefore, is to capture any references to local and absolute file paths and replace it during markdown processing. To enable this, I wrote a simple find/replace extension that can be loaded along with other markdown extensions during the processing chaing. It works by being fed a python-flavored regex string for the pattern to search for, along with a replacement string. We have to artfully insert this find-replace strategy, therefore, wherever we work with file paths, namely, in the contents of the markdown file, as well as any time we list files in the yaml preamble.

models.py

Open up homepage/blog/models.py and scroll below the imports at the top of the file. Add the following:

import re

import markdown_extensions.find_replace_pre

FILE_SEARCH=r"""(?P<group2>[a-zA-Z0-9./\-_ ]+(png|jpg|mp4|json|csv|xlsx|yml|trz))"""

ABSFIND_CORE=r"""^/"""+FILE_SEARCH
ABSREPLACE_CORE=settings.STATIC_URL+r'''\g<group2>'''

RELFIND_CORE="""^(?!/)"""+FILE_SEARCH
RELREPLACE1_CORE=r"""/"""
RELREPLACE2_CORE=r"""\g<group2>"""


ABSFIND=r"""(?P<group1>\]\(|src=")/"""+FILE_SEARCH+r"""(?P<group4>\)|"| *\n)"""
ABSREPLACE=r"""\g<group1>"""+settings.STATIC_URL+r"""\g<group2>\g<group4>"""

RELFIND=r"""(?P<group1>\]\(|src=")(?!/)"""+FILE_SEARCH+r"""(?P<group4>\)|"| *\n)"""
RELREPLACE1 = r'''\g<group1>/'''
RELREPLACE2 = r'''\g<group2>\g<group4>'''

These lines are some hard-coded regex strings that will help us find local and absolute paths to files in our existing markdown so we can replace them with the same static URL that Django replaces in its html templates for us.

Because relative links to files will no longer be pointing locally, we have to convert any relative file links we find into paths that know their location in their file structure. These links should be relative also to the post they are housed in, so we can add a new method, get_relpath() that will fill in that missing information.

def get_relpath(self):
    return os.path.split(self.path)[0]

We now need to update the render_markdown() method. It should look like this:

@staticmethod
def _render_markdown(input):
    md = markdown.Markdown(extensions=settings.MARKDOWN_EXTENSIONS)
    html_pass_1=md.convert(input)
    toc_tokens=md.toc_tokens
    my_dict = dict(html=html_pass_1,toc_tokens=toc_tokens)
    return my_dict

Replace the entire existing method with the following:

def _render_markdown(self,input):
    md_ex = settings.MARKDOWN_EXTENSIONS
    md_ex.extend([
        markdown_extensions.find_replace_pre.FindReplaceExtension(find=RELFIND,replace=RELREPLACE1+self.get_relpath()+'/'+RELREPLACE2,priority=202,name='rel-find-replace'),
        markdown_extensions.find_replace_pre.FindReplaceExtension(find=ABSFIND,replace=ABSREPLACE,priority=201,name='abs-find-replace'),
        ])
    md = markdown.Markdown(extensions=md_ex)
    html_pass_1=md.convert(input)
    toc_tokens=md.toc_tokens
    my_dict = dict(html=html_pass_1,toc_tokens=toc_tokens)
    return my_dict

In this case we added a few lines in order to add some find/replace filters to our markdown conversion process. These find/replace filters are informed by the regex strings we added at the top, and do two things:

1. Look for relative file paths to specific filetypes in our markdown directories, and convert them to absolute paths.
2. Look for absolute file paths to specific filetypes in any directory, and append the STATIC_URL to the front.

We finally return the updated html with return html_pass_2,toc_tokens

Add a top-level index.html

Inside homepage/blog/static/, create a new file called index.html.

Paste the following inside:

Unauthorized

Once our static url is up and running, this will serve to notify us that it is working, but also to mask any other information about the folder that might be provided by the server we choose.

New Static App

Here is the problem: I am starting from a mixed directory structure consisting of both markdown files and supporting media files. With my old site in Hugo, if I wanted to link to an image in the same directory I could use a relative link. The advantage was that I didn't need to duplicate the structure of my site, but could locally link images to files in the same directory. Thus, when moving or reorganizing directories, I wouldn't break links to locally-linked images in the same folder. While acknowledging that each person's organizational strategy is different, I need to accommodate the possibility that you might also prefer that file structure.

Now add this point: Django already has a feature for scanning extra directory structures and collecting those files into one place for serving using their split dynamic / static strategy. Great! So what is the problem?

Well, it is a mixed folder structure, consisting of markdown AND media files. We already created the import script that scans through that same file structure, identifies markdown files, and imports their content into the database. If I simply point the collectstatic script to this same directory, it will suck up all the files in that folder structure, including the origingal markdown source. Thus, I need a way to scan and collect all media files while leaving the original markdown files where they are. Sure, I could write a custom bash or python script to temporarily collect everything first, but does Django have it's own approach? Yes, we need to modify the way collectstatic works, though

I used this approach:

• https://docs.djangoproject.com/en/6.0/ref/contrib/staticfiles/#customizing-the-ignored-pattern-list
• https://stackoverflow.com/questions/78724509/unknown-command-collectstatic-after-overriding-staticfilesconfig

Alternatives could also include implementing your own static file finder, as discussed here:

• https://nextgendjango.com/managing-static-files-in-django-projects-a-practical-guide.html#debugging-common-static-files-issues-diagnosing-and-resolving-frequent-problems
• https://stackoverflow.com/questions/12082902/how-do-i-ignore-static-files-of-a-particular-app-only-with-collectstatic

apps.py

create a new file in homepage/staticfiles/apps.py and paste in the following code:

from django.contrib.staticfiles.apps import StaticFilesConfig

class StaticFilesConfig(StaticFilesConfig):
    name = 'staticfiles'
    ignore_patterns = [
        "CVS", ".*", "*~",
        "*.md",
        ]

staticfiles.py

create a new file at homepage/staticfiles/staticfiles.py and paste in the following:

import django.contrib.staticfiles.apps

class StaticFilesConfig(django.contrib.staticfiles.apps.StaticFilesConfig):
    ignore_patterns =  ['CVS', '.*', '*~','*.md'],

collectstatic.py, findstatic.py, and runserver.py

Even though this new app will now be filtering out specific files (such as Markdown), We still want it to behave normally with regard to other functions provided by our project's manage.py. Therefore, we need to create pass-through modules, in the form of three new scripts. These scripts will simply allow our new app to work the same as the default static file app provided by django, by connecting to its built-in functions.

Create three new files. The first one should be homepage/staticfiles/management/commands/collectstatic.py, with the following code pasted in:

from django.contrib.staticfiles.management.commands.collectstatic import Command

The second one should be homepage/staticfiles/management/commands/findstatic.py, with the following code pasted in:

from django.contrib.staticfiles.management.commands.findstatic import Command

The third one should be homepage/staticfiles/management/commands/runserver.py, with the following code pasted in:

from django.contrib.staticfiles.management.commands.runserver import Command

Create __init__.pyfiles

Create three blank files named __init__.py and place them in the following places:

• homepage/static/__init__.py
• homepage/static/management/__init__.py
• homepage/static/management/commands/__init__.py

homepage/homepage/settings.py

Find the list that defines INSTALLED_APPS. Comment out the line with 'django.contrib.staticfiles' and add a new line:

'staticfiles.apps.StaticFilesConfig',

Your final list of apps should look like this:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    # 'django.contrib.staticfiles', #removed due to update below
    #added by me
    'staticfiles.apps.StaticFilesConfig',
    'blog.apps.BlogConfig',
]

In homepage/homepage/settings.py look for the line with STATIC_URL. For now, keep STATIC_URL = 'static/' the same, but add the following lines below it:

STATIC_ROOT = '/absolute/path/to/your/collected_files'
STATICFILES_DIRS = [
    '/absolute/path/to/your/markdown/content/directory',
    '/absolute/path/to/your/markdown/static/directory',
    ]
# custom addition for defining the search path for markdown file imports
MARKDOWN_SOURCE_PATH = '/home/danaukes/repos/websites/danb0b.github.io/content'

With this example, replace /absolute/path/to/your/collected_files with the location where you would like Django's collectstatic function to put all collected static files. Next, replace /absolute/path/to/your/markdown/content/directory with the absolute path to the /content directory for your markdown file source. Finally, replace /absolute/path/to/your/markdown/static/directory with the absolute path to the /static directory often used to separate markdown content from static files. Coming from hugo, these are the /content and /static directories directly under my top-level Hugo project folder.

Collect Static