Lecture: Importing Markdown Content

Importing Markdown Content

Create init.py

Create the following two directories and add a blank text file named __init__.py to each:

• homepage/blog/management
• homepage/blog/management/commands

Create Import Script

Next we are going to actually create the script that imports markdown files and saves them as individual Post entries in the underlying database. Add a new module at homepage/blog/management/commands/import_markdown.py and paste in the following to import all the libraries we will need:

#!/usr/bin/env python3
import os
from django.core.management.base import BaseCommand
from django.core.exceptions import ValidationError
from django.utils import timezone
from blog.models import Post
from blog.models import Folder
from blog.models import User
from blog.models import Tag
from django.conf import settings
import glob
import datetime
import pytz
import json
import frontmatter

Next we are going to create a function that fixes a path it gets by removing the user shorthand notation, expanding any relative paths to absolute, fixing the case of any characters, and fixing any character irregularities

def fix_path(path_in):
    path = path_in
    path = os.path.expanduser(path)
    path = os.path.abspath(path)
    path = os.path.normcase(path)
    path = os.path.normpath(path)
    return path

Next we will create a function that generates a list of Tags starting from a list of tag names. First, the function creates a list of all existing tags in the database and then returns the lowercase name of each as a string. Then, given a list of tag names it either generates a new Tag or adds the existing Tag to a new list of Tags. That list of Tags is then returned.

def create_tags(tag_list):
    tags = Tag.objects.all()
    tag_names = [tag.name.lower() for tag in tags]
    post_tags = []
    for tag_name in tag_list:
        if tag_name.lower() not in tag_names:
            tag = Tag(name=tag_name)
            tag.full_clean()
            tag.save()
            post_tags.append(tag)
        else:
            post_tags.extend(Tag.objects.filter(name=tag_name))
    return post_tags

The next function will update a post with new information. First, given a post, it will:

1. Print the current path
2. Compose the full path out of the relative path and the MARKDOWN_SOURCE_PATH found in django's settings
3. Load the markdown "front-matter" into a dict object. Front matter is the data stored in the yaml preamble of the markdown file.
4. Convert that front matter to a dictionary
5. Create a list of tags from the dictionary's list of tags, if present
6. Extract the content and then delete the "content" key from the dictionary, as we will store that separately.
7. Convert the dictionary to a json string
8. Set the author to the primary Django user (with primary key of 1)
9. Set the post's json variable to the front-matter data extracted from the markdown file, in json format
10. Set the content of the post from the content of the dictionary
11. Set the folder as the Folder element
12. Update the date, converting it to a localized timezone in the process
13. Validate the changes
14. Save the post
15. Update the tags with the list of tags created earlier. This has to come after saving for connections between items with a many-to-many relationship.

def update_post(post,parent):
    path = post.path
    print(path)

    fullpath=os.path.join(settings.MARKDOWN_SOURCE_PATH,path)

    with open(fullpath) as f:
        fm = frontmatter.load(f)

    my_dict = fm.to_dict()
    try:
        tags = create_tags(my_dict['tags'])
    except KeyError:
        tags = []
    content = my_dict['content']
    del my_dict['content']
    my_json = json.dumps(my_dict,default=str)
    post.author = User.objects.get(pk=1)
    post.json = my_json
    post.content = content
    post.folder = Folder.objects.get(name=parent)

    try:
        my_tz = pytz.timezone(settings.TIME_ZONE)
        date = my_dict['date']
        dt = datetime.datetime.combine(date, datetime.datetime.min.time())
        post.date = my_tz.localize(dt)
    except KeyError:
        pass

    try:
        post.full_clean()
    except ValidationError as e:
        raise

    post.save()
    post.tags.set(tags)

Finally we get to the import_markdown() function. This function is responsible for

1. Scanning through a directory path
2. Identifying all the markdown files in the path
3. Constructing a folder:files dictionary
4. Deleting and then recreating all the folders in the database
5. Identifying posts that are new, posts to be updated, and posts to be deleted.
6. Deleting Posts whose files have been deleted
7. Updating Posts that already exist
8. Creating new Posts for new markdown files.

For the completely annotated function, I recommend visiting the git repository, whose code is fully commented:

def import_markdown(my_path):

    my_path = fix_path(my_path)

    markdown_file_structure = {}

    for d,sd,f in os.walk(my_path):
        fixed_path=fix_path(d)
        rel_dir = os.path.relpath(fixed_path,my_path)
        markdown_files = glob.glob(os.path.join(fixed_path,'*.md'))
        markdown_files = [os.path.relpath(os.path.join(fixed_path,item),my_path) for item in markdown_files]
        terminal_directory=False
        if len(markdown_files)==1:
            if os.path.split(markdown_files[0])[1]=='index.md':
                markdown_file_structure[os.path.split(rel_dir)[0]].extend(markdown_files)
                terminal_directory=True

        if not terminal_directory:
            if rel_dir=='.': rel_dir=''
            markdown_file_structure[rel_dir] = markdown_files

    for folder in Folder.objects.all():
        folder.delete()

    print(markdown_file_structure.keys())

    for item in markdown_file_structure.keys():
        folder=Folder(name=item)
        folder.full_clean()
        folder.save()

    file_structure_inv = {}
    for folder, files in markdown_file_structure.items():
        file_structure_inv |= dict([(file,folder) for file in files])

    markdown_relative_paths = []
    for key,value in markdown_file_structure.items():
        markdown_relative_paths.extend(value)

    existing_paths = [post.path for post in Post.objects.all()]

    dead_paths = sorted(list(set(existing_paths) - set(markdown_relative_paths)))
    new_paths=sorted(list(set(markdown_relative_paths) - set(existing_paths)))
    update_paths = sorted(list(set(existing_paths) & set(markdown_relative_paths)))

    for post in dead_paths:
        post=Post.objects.get(path=post)
        post.delete()

    for item in new_paths:
        folder = file_structure_inv[item]
        post = Post(path=item)
        update_post(post,folder)

    for item in update_paths:
        folder = file_structure_inv[item]
        post = Post.objects.get(path=item)
        update_post(post,folder)


    print(len(dead_paths),' dead posts deleted')
    print(len(new_paths),' new posts added')
    print(len(update_paths),' posts updated')

Finally, we need to connect the import_markdown() function to Django's management command class. This allows us to call the import_markdown() function from the manage.py command-line tool. Add the following lines to the bottom of the file:

class Command(BaseCommand):
    help = 'Displays current time'
    def handle(self, *args, **kwargs):
        import_markdown(settings.MARKDOWN_SOURCE_PATH)

admin.py

In addition to the command line interface, we also want to make it possible to import markdown files directly from the admin interface. This will require connecting to some of Django's existing functionality in the administration project. Open up homepage/blog/admin.py and import the following:

from .management.commands.import_markdown import import_markdown

This will bring in new functionality from the management.commands module that we are about to create. Next let's update PostAdmin. We will add some new data and methods. First, we are going to define a new template path to use, change_list_template, along with two new methods, get_urls() and update_post_view().

class PostAdmin(admin.ModelAdmin):
    change_list_template = 'admin/blog/post/change_list.html'

    def get_urls(self):
        urls = super().get_urls()
        custom_urls = [
            path('update-post/', self.admin_site.admin_view(self.update_post_view), name='update-post'),
        ]
        return custom_urls + urls

    def update_post_view(self, request):
        context = {
            'title': 'Import Markdown Data',
            'opts': self.model._meta,
            'app_label': self.model._meta.app_label,
        }
        import_markdown(settings.MARKDOWN_SOURCE_PATH)
        return HttpResponseRedirect("../")

get_urls() conncets a new path to the administration portal's post type. if you go to https://localhost:8000/admin/post/update-post, it will connect to this new path and trigger the update_post_view() function directly below.

update_post_view is a view function that executes whenever reached at the update-post path we just defined. We add some boilerplate context variables indicated by this reference(https://www.lune.dev/questions/9016/how-can-i-add-a-custom-django-admin-button-to-import-json-data-for-creating-or-u) and then run the new import_markdown() function we imported at the top.

Create change_list.html

We need to add the button to the post's list of buttons. Create a new file at homepage/blog/templates/admin/change_list.html and paste in the following:

{% extends "admin/change_list.html" %}
{% load i18n admin_urls %}

{% block object-tools-items %}
    <li>
        <a href="{% url 'admin:update-post' %}">
            {% trans 'Import Markdown Files' %}
        </a>
    </li>
    {{ block.super }}
{% endblock %}

This template will add to the existing object-tools-items block. We see a new list item that creates an anchor with the admin:update-post keyword. This will run the import_markdown() function. the trans tag tells the template what to name the new button.

settings

Finally, we need to modify homepage/homepage/settings.py. To the bottom of this file, add:

MARKDOWN_SOURCE_PATH = '<root-path>'

In this case, you need to replace <root-path> with the absolute path to your markdown file tree on your computer. This tree should be a single parent folder with only subfolders beneath it. Underneath those child folders, you can have any arrangement of markdown files, images and other static content, and other subfolders.

root-path:
- directory1/:
  - subdir1/:
    - markdownfile1.md
    - image1.jpg
  - markdownfile2.md
  - movie.mp4
- directory2/:
  - subdir2/:
    - markdownfile3.md
    - image2.jpg
  - subdir3/:
    - index.md

Remember that any subdirectory with only one markdown file in it called index.md (as in subdir3/ in the previous example) will be treated as a post rather than as a list of posts.

edit post

Next, run your development server so that you can access the administrative interface. From homepage/ run it with the following command:

./run

1. Go to https://localhost:8000/admin/post
2. Edit notebook/test/post1.md

3. To the new 'json' field add:

   {"title":"hello"}
   

4. To the 'date' field, select "today" for the date and "now" for the time

5. You can keep the same content as before:

   ## Heading
   
   > This is a quote block
   
   
   This is some text we inserted.  Here is a raw  link: <https://danaukes.com>
   
   This file is located [at this link](http://localhost:8000/notebook/test/post1)
   

6. Hit save

What do we see when we hit run? Go back to https://localhost:8000/notebook/test/post1/. We see only a few changes, as most of our changes were to add new data and import them automatically, but we see the title ("hello") is now being used instead of the filename, but this title is derived straight from the get_title() function that draws its data from the json object. We also see the page title in the tab has been updated with the new title information as well.

Test Import from CLI

Now we're going to test importing our markdown files from the command line interface. Stop the running server by typing ctrl+c at the same time to kill the process from the terminal it was running in. Next, type

python manage.py import_markdown

and re-run your server with:

./run

On your administration screen, check out the new list of posts:

Navigate to https://localhost:8000/posts. You should now see

More importantly, click on one of your posts. At the top you should see the directory structure as part of each page. Clicking on a directory should allow you to see a list of posts in that directory, as well as the parent and child folders. Already, we have built-in navigation of our site!

In addition to a customized list of posts, we see that the title of this page changes to accommodate the folder's name ("Posts in ..."). Even though we only had one template, it can be used over and over with the right tags and variables.

Test Button