Last week I created another Redmine plugin to setup staging and testing servers with Redmine called Redmine Remove Membership. It's a simple plugin that adds a Rake task to Redmine that will revoke the membership of a user from all projects. My client is using this on their staging server to quickly setup a limited test environment for their users.

Download

The plugin can be download from the Little Stream Software project or from GitHub.

Help

If you need help, my Redmine bug tracker is open to the public and you are welcome to ask for help there.

This past week I created a plugin for one of my Redmine clients called Redmine Lock Users. It's a simple plugin that adds a Rake task to Redmine to make it easy to lock all of the user accounts in the system.

We have been using this to lock down testing and staging servers. Since those servers have the same data as the production Redmine, any user could log into them on accident and not realize it. With this plugin we can lock all users accounts and only allow access for specific users.

Download

The plugin can be download from the Little Stream Software project or from GitHub.

Help

If you need help, my Redmine bug tracker is open to the public and you are welcome to ask for help there.

Upgrading Redmine is a pretty standard process. But with the major 1.0 release and over a hundred plugins for Redmine now, it can be a daunting task.

If you have a problem upgrading and need help now, I have a single support service for sale that can be used to upgrade your Redmine. If that's not enough or you have a misbehaving plugin that needs to be fixed, I also have a Custom Development service for sale.

Eric Davis

A bug fix release for Redmine Simple Support has just been posted to my Redmine. This will fix a compatibility issue with the Redmine 0.9 releases.

Download

The plugin can be download from the Little Stream Software project or from GitHub.

Features

• Compatibility With 9.0-Stable Branch

Help

If you need help, my Redmine bug tracker is open to the public and you are welcome to ask for help there.

Thanks

I would like to thank Jean-Baptiste Barth for reporting the bug and researching the fix.

I'm going to be pausing my daily code reading posts this week. Between them and my refactoring series, I've been close to posting every day for 2010 and I need a short break.

Redmine 1.0.0 has been released!!!

That is all.

In this weeks code readings, I've taken a deep dive tour into how Redmine formats it's "wiki" text. Today I'm going to wrap it up a final look at the wiki macro execution, #exec_macro.

Review

Remember, here is the example macro I'm using:

This is a page that will include Design

{{include(Design)}}

textilizable

text = Redmine::WikiFormatting.to_html(Setting.text_formatting, text, :object => obj, :attribute => attr) { |macro, args| exec_macro(macro, obj, args) }

Starting inside of #textilizable's #to_html block:

• macro gets set to 'include'
• 'obj' is the object passed into #textilizable, like a wiki page
• and args is an array ['Design']

exec_macro

module Redmine
  module WikiFormatting
    module Macros
      module Definitions

        def exec_macro(name, obj, args)
          method_name = "macro_#{name}"
          send(method_name, obj, args) if respond_to?(method_name)
        end

      end
    end
  end
end

#exec_macro is really simple. It creates a method name by prefixing "macro_" to the name ("macro_include") and then uses send to call that method with the object and macro arguments. (send('macro_include', some_wiki_page, ['Design'])).

In order to see how the macro_include method was defined, we need to look at the rest of the macro API (it's small).

macro

# Defines a new macro with the given name and block.
def macro(name, &block)
  name = name.to_sym if name.is_a?(String)
  @@available_macros[name] = @@desc || ''
  @@desc = nil
  raise "Can not create a macro without a block!" unless block_given?
  Definitions.send :define_method, "macro_#{name}".downcase, &block
end

The #macro method is used to define a new macro in Redmine. It's given a name, a block, and is added to the @@available_macros data structure. The key to this method is the :define_method on the last line:

Definitions.send :define_method, "macro_#{name}".downcase, &block

This will dynamically define a method based on the macro name that calls the macro's block as the method body. Remember, Definitions is the module above with #exec_macro.

To wrap out this tour of the macros, lets take a look at the actual include macro now.

include macro

macro :include do |obj, args|
  page = Wiki.find_page(args.first.to_s, :project => @project)
  raise 'Page not found' if page.nil? || !User.current.allowed_to?(:view_wiki_pages, page.wiki.project)
  @included_wiki_pages ||= []
  raise 'Circular inclusion detected' if @included_wiki_pages.include?(page.title)
  @included_wiki_pages << page.title
  out = textilizable(page.content, :text, :attachments => page.attachments)
  @included_wiki_pages.pop
  out
end

The first thing this macro does is uses the args to find the page name ('Design'). Wiki#find_page supports prefixing pages with a project to do cross project links.

  @included_wiki_pages ||= []
  raise 'Circular inclusion detected' if @included_wiki_pages.include?(page.title)
  @included_wiki_pages << page.title

@included_wiki_pages is used to keep track of all of the pages included. This will make sure that one page doesn't include another page, which includes the first page, which includes the second page, which includes the first... well, you get the point. It's circular.

  out = textilizable(page.content, :text, :attachments => page.attachments)
  @included_wiki_pages.pop
  out

Then the macro runs #textilizable on the page content and returns the output. This lets the include macro act recursively, each page that gets included can include other pages until the including stops or a circular inclusion occurs.

That wraps up this week's code reading on Redmine's formatting. I covered how content is sent to the formatting system, how Redmine decides which formatter to use, the details of the Redmine Textile formatter, how macros are detected, and finally how a macro is defined and executed. I didn't get a chance to go through Redmine's custom syntax but maybe I'll tackle that with a later series.

Today I'm digging back into the WikiFormatting to read through #execute_macros.

The Code

module Redmine
  module WikiFormatting
    class << self
      MACROS_RE = /
                    (!)?                        # escaping
                    (
                    \{\{                        # opening tag
                    ([\w]+)                     # macro name
                    (\(([^\}]*)\))?             # optional arguments
                    \}\}                        # closing tag
                    )
                  /x unless const_defined?(:MACROS_RE)
      
      # Macros substitution
      def execute_macros(text, macros_runner)
        text.gsub!(MACROS_RE) do
          esc, all, macro = $1, $2, $3.downcase
          args = ($5 || '').split(',').each(&:strip)
          if esc.nil?
            begin
              macros_runner.call(macro, args)
            rescue => e
              "<div class=\"flash error\">Error executing the <strong>#{macro}</strong> macro (#{e})</div>"
            end || all
          else
            all
          end
        end
      end


    end
  end
end

Review

It will be best to use an example macro in order to trace the flow of this method. Redmine has a macro called include which can be used to include another wiki page into the current one. Lets use the following text as our page content:

This is a page that will include Design

{{include(Design)}}

Tracing the flow of this macro, the call stack looks like:

1. ApplicationHelper#textilizable calls Redmine::WikiFormatting.to_html with a block { |macro, args| exec_macro(macro, obj, args) }
2. Redmine::WikiFormatting.to_html uses the configured formatters to format the plain text
3. Since Redmine::WikiFormatting.to_html was passed a block, it runs execute_macros(text, block) which passes the block from #textilizable

Now the program ends up in the method above with the converted text and the block from #textilizable. At this point it runs the MACROS_RE regular expression through gsub! so it can replace all of the macros with their content.

MACROS_RE regular expression

      MACROS_RE = /
                    (!)?                        # escaping
                    (
                    \{\{                        # opening tag
                    ([\w]+)                     # macro name
                    (\(([^\}]*)\))?             # optional arguments
                    \}\}                        # closing tag
                    )
                  /x unless const_defined?(:MACROS_RE)

The first thing to notice is that the expression uses the /x modifier. This tells it to ignore whitespace between each regular expression token and lets the expression be written on multiple lines with inline comments. Without it, the expression would be much harder to read:

/(!)(\{\{([\w]+)(\(([^\}]*)\))?\}\})/

I think the comments explain what parts match but here are the different match sets:

1. the "!" prefix to escape
2. the entire macro after the escape
3. the macro name, without the {}
4. all of the arguments enclosed with ()
5. all of the arguments

gsub

Having done a lot of regular expression work on my book, I've become very familiar with how regular expressions are used with #gsub! and a block. First #gsub! checks if the text matches and for every match it yields to the block, setting up the standard $1, $2, $n variables from the regular expression result. Whatever is returned from #gsub! is used as the replacement.

For example:

"Little Stream Software writes Rails code".gsub!(/(Rails)/) do
  "Redmine"
end

# returns => "Little Stream Software writes Redmine Code"

So when the MACROS_RE matches, the regular expression results are setup and passed into the block.

Macro Arguments

args = ($5 || '').split(',').each(&:strip)

MACROS_RE matches the arguments and passes them down as a comma separated string; 'Design' from the example. || is used in case the macro wasn't passed any arguments. Then split is used to convert the string to an array of arguments, separated by the commas. Finally, strip is called on them to handle any extra whitespace.

Escaped macro

          if esc.nil?
            # ...
          else
            all
          end

It took me a minute to understand what esc was doing but then I remembered that it's only set when the macro is escaped so it can be printed to the page. So when the macro is escaped, the full macro content is rendered using the second match ($2 or all).

Calling the macro

            begin
              macros_runner.call(macro, args)
            rescue => e
              "<div class=\"flash error\">Error executing the <strong>#{macro}</strong> macro (#{e})</div>"
            end || all

At this point we have a macro (include) and some args (['Design']). So the block passed all the way down from textilizable is run and the output is passed back up to #to_html.

There is still one more level of code that I need to dig though before we get to the actual running macro. This is in the exec_macro method that textilizable uses.

After reading the NullFormatter yesterday, I wanted to take a short look at how Redmine's Textile formatter works. I'm going to try to avoid much of the parsing code as possible, it's very complex and could use an entire week of code reading itself.

The Code

require 'redcloth3'

module Redmine
  module WikiFormatting
    module Textile
      class Formatter < RedCloth3
        include ActionView::Helpers::TagHelper
        
        # auto_link rule after textile rules so that it doesn't break !image_url! tags
        RULES = [:textile, :block_markdown_rule, :inline_auto_link, :inline_auto_mailto, :inline_toc]
        
        def initialize(*args)
          super
          self.hard_breaks=true
          self.no_span_caps=true
          self.filter_styles=true
        end
        
        def to_html(*rules)
          @toc = []
          super(*RULES).to_s
        end
  
      private
  
        # Patch for RedCloth.  Fixed in RedCloth r128 but _why hasn't released it yet.
        # <a href="http://code.whytheluckystiff.net/redcloth/changeset/128">http://code.whytheluckystiff.net/redcloth/changeset/128</a>
        def hard_break( text ) 
          # ... snip ...
        end
        
        # Patch to add code highlighting support to RedCloth
        def smooth_offtags( text )
          # ... snip ...
        end
        
        # Patch to add 'table of content' support to RedCloth
        def textile_p_withtoc(tag, atts, cite, content)
          # ... snip ...
        end
  
        alias :textile_h1 :textile_p_withtoc
        alias :textile_h2 :textile_p_withtoc
        alias :textile_h3 :textile_p_withtoc
        
        def inline_toc(text)
          # ... snip ...
        end
        
        # Turns all urls into clickable links (code from Rails).
        def inline_auto_link(text)
          # ... snip ...
        end
  
        # Turns all email addresses into clickable links (code from Rails).
        def inline_auto_mailto(text)
          # ... snip ...
        end
      end
    end
  end
end

Review

The first thing to notice is that Textile::Formatter is a subclass of RedCloth3. This will let the formatting reuse many of the methods from RedCloth directly. Since the formatter is used by calling #new and then #to_html, I'll look at those methods first.

initialize

def initialize(*args)
  super
  self.hard_breaks=true
  self.no_span_caps=true
  self.filter_styles=true
end

RedCloth's #initialize takes two arguments: string and restrictions. Redmine only uses the first one, passing in the text content that needs to be formatted. After RedCloth has initialized the object, Formatter sets three attributes to control the rendering:

• hard_breaks - converts single newlines to HTML break tags.
• no_span_caps - turns off the behavior where capitalized words automatically have span tags placed around them.
• filter_styles - turns off textile styles. I know this is off because there have been many security bugs found when styles were enabled

Now with a fresh object and the text content, #to_html is used to... well, convert the text into HTML.

to_html

def to_html(*rules)
  @toc = []
  super(*RULES).to_s
end

Redmine's textile supports creating a table of contents from a wiki "macro" (1). The @toc variable will be used later in the #inline_toc method to render the table of contents.

Other than that, Formatter just calls super with the formatting rules to apply:

• textile - provided by RedCloth, this runs the actual Textile conversion.
• block_markdown_rule - provided by RedCloth, this is a single rule from RedCloth's markdown support. It converts Markdown's horizontal rules into an <hr> element.
• inline_auto_link - provided by Redmine, this converts all urls into clickable links.
• inline_auto_mailto - provided by Redmine, this converts all mailto urls into clickable links.
• inline_toc - provided by Redmine, this creates the automatic table of contents from the HTML header tags (h1, h2, h3).

Finally after all of the formatting is complete, Formatter converts the content to a string and returns if back up the stack.

So these two methods are all that's needed to hook up Redmine to RedCloth. I did some work on the Redmine markdown plugin and it also had to implement these methods to connect to a markdown library.

(1) Technically toc isn't a macro though there will be a big push to port it to Redmine's macro system.

Yesterday's post showed how ApplicationHelper#textilizable used Redmine::WikiFormatting#to_html to convert the text content into HTML.

text = Redmine::WikiFormatting.to_html(Setting.text_formatting, text, :object => obj, :attribute => attr) { |macro, args| exec_macro(macro, obj, args) }

Today I'm going to look into the #to_html method to see how the conversion is run.

The Code

module Redmine
  module WikiFormatting
    class << self

      def to_html(format, text, options = {}, &block)
        text = if Setting.cache_formatted_text? && text.size > 2.kilobyte && cache_store && cache_key = cache_key_for(format, options[:object], options[:attribute])
          # Text retrieved from the cache store may be frozen
          # We need to dup it so we can do in-place substitutions with gsub!
          cache_store.fetch cache_key do
            formatter_for(format).new(text).to_html
          end.dup
        else
          formatter_for(format).new(text).to_html
        end
        if block_given?
          execute_macros(text, block)
        end
        text
      end
    end
  end
end

Review

The first thing that #to_html does is to check if the formatted text has been cached in ActiveSupport::Cache. Then it uses #formatter_for which will pick the configured formatter to render the content. Finally, the Redmine macros are run with #execute_macros.

Caching

text = if Setting.cache_formatted_text? && text.size > 2.kilobyte && cache_store && cache_key = cache_key_for(format, options[:object], options[:attribute])
  # Text retrieved from the cache store may be frozen
  # We need to dup it so we can do in-place substitutions with gsub!
  cache_store.fetch cache_key do
    formatter_for(format).new(text).to_html
  end.dup

Caching the text formatting was a recent addition to Redmine, so it's still very strict about when content is cached.

• The Cache Formatted Text setting in the Administration panel needs to be enabled, and
• The text size needs to be bigger than 2K, and
• The cache check should miss (i.e. no preview content cached)

When all of these cases match, Redmine will run the block passed to cache_store.fetch and store the result into the cache for later use. Then Redmine uses the #formatter_for method to render the content.

Content rendering

def formatter_for(name)
  entry = @@formatters[name.to_s]
  (entry && entry[:formatter]) || Redmine::WikiFormatting::NullFormatter::Formatter
end

Redmine::WikiFormatting#to_html calls formatter_for(format).new(text).to_html both when caching the content and when no caching is enabled. formatter_for is just using the lookup table of valid formatters and returning their class to the caller. What's nice about this process is that Redmine is able to return the NullFormatter if nothing is found, which gives a good plain text fallback. If I assume that the NullFormatter is used, then the call for formatter_for would be converted to this:

Redmine::WikiFormatting::NullFormatter::Formatter.new(text).to_html

NullFormatter#to_html

# Default formatter module
module NullFormatter
  class Formatter
    include ActionView::Helpers::TagHelper
    include ActionView::Helpers::TextHelper
    include ActionView::Helpers::UrlHelper
        
    def initialize(text)
      @text = text
    end
        
    def to_html(*args)
      simple_format(auto_link(CGI::escapeHTML(@text)))
    end
  end
      
  module Helper
    def wikitoolbar_for(field_id)
    end
      
    def heads_for_wiki_formatter
    end
      
    def initial_page_content(page)
      page.pretty_title.to_s
    end
  end
end

To get a better idea of what a Formatter does, here is the entire NullFormatter::Formatter class. It's easy to see that there isn't very much going on here. First a new object is initialized with the text that needs to be rendered. Then #to_html uses simple_format and auto_link to create a basic HTML section.

Macro execution

if block_given?
  execute_macros(text, block)
end

Finally Redmine::WikiFormatting#to_html runs #execute_macro when there is a block argument. I'll save reviewing #execute_macros until later, it's a complex method that uses a large regular expression to match the macros in the text.

Now that I have a better understanding of how Redmine's Wiki Formatting works, I'm going to take a deeper look at how the textile formatter is structured. Since it's using _why's RedCloth3 I'm expecting this piece to be pretty complex.