Re: [DIYbio] Re: Standardized DIYbio report format? WAS: Endophyte isolation and first successful sequencing

Hi all,
Inspired by the notion, I've partially written a python script that
implements the idea of a "Markup language for Academic Authorship".

It's not ready, it won't dump a paper for you yet, but it has most of
the basics ready:
https://gitorious.org/plod/plod

Included is an unfinished "paper" for demonstration purposes. If you
compile, you'll get a simple layout that demonstrates citation
formatting, but you'll only notice the abstract extension at work if
you look at the html and note the class and id attributes.

The gist:
0. It's not ready, so seriously; don't bother if you're not interested
in hacking on the code and making it ready.
1. It's Extended Markdown, using the Markdown package from PyPI
(required): https://pypi.python.org/pypi/Markdown/2.3.1
2. It has (currently) two custom extensions:
    - one is a "fencing" scheme for defining the abstract as a
      separate, named HTML block so it can be styled and laid out
      differently: The abstract is surrounded in "@@@" (@=a=Abstract)
      to achieve this.
    - the other is a citation format like what I wrote up earlier in
      this thread. If this format is desired (you can always do it
      manually), it is written as "{{ <fieldname> <contents...> |
      <field2> <contents...> }}". To actually place citations just use
      the "footnotes" extension as included in Extended Markdown;
      follow the cited statement with "[^tag]", then define later with
      "[^tag]: {{ <citation block> }}". The tag can be anything
      memorable; it will be replaced with a number in order of
      occurrence. The citation block will be placed either at the end
      of the document, or wherever "///References List///" is found.
3. The "metadata" extension allows definitions of metadata at the very
beginning of the document, which is not rendered into the final
document but alters its rendering. At present, only "title" is used; it
becomes the HTML page title, and until a better way to automatically
detect the paper title is cooked up, this is the only way.
4. The "attributes" extension is enabled, and is used to set the
class/id of some elements that vary between papers, like the title;
you'll see in the example "test1.md" document that the title is
followed by {: #paper-title }, an ugly format used by the extension to
add "paper-title" as the HTML ID of the paper title. A postprocessor
extension might resolve the page-title issue more cleanly but would
require some sort of standard title formatting anyway to distinguish it.

Other than the above, it's standard extended markdown; you format the
same way and you process the same way. It compiles to HTML and can be
embedded in a template (read the script to see how the template works,
if given). Indeed, that's the whole point; right now, it just compiles
to a flat HTML file, but with the right HTML/CSS template, it could
compile to a properly formatted "paper" as you might read on PLoS or
Pubmed.

A lot of nice things remain that could be implemented better. I would
like to see these features:
1. Framed images, for figures with explanatory subtext.
2. Insertion of other markdown documents, allowing a source file to be
separated, perhaps so important tables can be handled independently and
compiled into the main source as needed. Would also be handy for larger
works, to be broken into chapters.
3. "Cited by", though this would probably require a central database of
citations to check with, which is pretty out of theme with
peer-produced academic literature.

On Fri, 16 Aug 2013 14:14:35 +0100
Cathal Garvey <cathalgarvey@cathalgarvey.me> wrote:

> My "criticisms" of wiki were directed at "wiki-only", not "wiki as
> well".
>
> In other words, if your files are written in a standard,
> machine-parseable format that's accessible somewhere, then rendering
> and displaying on a wiki is nice for human navigation between papers
> and references (although with proper linking within pages it shouldn't
> matter whether they're all hosted on the same site/wiki.), and
> provides a visual set of modifications to the file over time.
>
> However, the "wiki" is little better than a Wordpress install for this
> purpose; the usual purpose of a wiki is to be human-editable, but if
> it's merely a rendering of a source file stored elsewhere, then
> changes made to the wiki will not be passed back to the file.
>
> But, having "wiki-only", while having the advantage that it's
> human-editable (preferably by author/editor only!) and the
> presentation *is* the file, has the disadvantage that it's poorly
> machine parseable, making conversion into other formats and analysis
> less friendly to meta-study and development. The machine-parseable
> "wikisource" is usually stored as a database entry somewhere rather
> than as a readily-accessible file to a passing bot.
>
> Also, wikitext, while originally as simple as Markdown, has sort of
> devolved a bit, and has some non-obvious syntax which means lots of
> checking syntax guides during authorship: does the bar character come
> before the link or after the link? How do you do alt-text again?
> References ditch the normal syntax entirely and look like a horrid mix
> of wikitext and html.
>
> So, I'm not a big fan, but it's still far
> friendlier, machine-parseable and standardised than most other
> alternatives. Even LaTeX is terrible these days; it appears the
> primary goal of LaTeX developers over time has been to guarantee
> backwards compatibility rather than to develop a great markup format.
> Doing anything useful requires ten or more plugins which don't come as
> standard with LaTeX. The "development environment" for new authors is
> forbidding. While Markdown is poorly standardised too, it comes with
> more batteries and compiles optionally to basic LaTeX if you really
> want to do hardcore layout work.
>
> I'm contemplating devoting an evening into making a "scientific
> article compiler" in Python for Markdown; any suggestions on what
> you'd like to see most in an application like this? Or how you'd like
> to see it implemented? I'm assuming that the default presentation
> format these days is HTML but will look into creating a layout that
> can be printed to a niceish PDF, too.
>
> On Thu, 15 Aug 2013 12:33:44 -0700
> Nathan McCorkle <nmz787@gmail.com> wrote:
>
> > That seems like a good list of 'wants' for what a software for this
> > might be.
> >
> > I disagree about your wiki coment... It's stored as markdown but
> > rendered as html, why would you try to parse rendered HTML? It's not
> > cloud only, you can definitely run a local wiki, or download the
> > markdown and render it in some custom LaTex prettifier or something
> > else. On Aug 14, 2013 7:02 AM, "Cathal Garvey"
> > <cathalgarvey@cathalgarvey.me> wrote:
> >
> > > The wiki parses Wikitext, yes, but the actual rendered wiki is
> > > standard HTML and isn't designed explicitly for ease of parsing.
> > >
> > > I'm also against relying on Wiki infrastructure because wikis are
> > > "cloud only", in that rendering a wiki page to a local file is
> > > often difficult, because of relative pathnames and image embeds,
> > > database backed infrastructure, etc. - Yes, you *can* export to
> > > PDF, but PDF is effectively a read-only data structure.
> > >
> > > ## Requirements for Ease and Academic Rigour
> > > I think the ideal is a format that:
> > > * Is easy for people to _read_.
> > > * Is easy for people to _write_, requiring minimal reference and a
> > >   shallow learning curve.
> > > * Is already in widespread use if possible, so people have little
> > > to learn and plenty of support when they do.
> > > * Allows image embedding.
> > > * Allows dynamic styling, so it can be rendered into a pretty
> > >   "Proceedings of DIYbio" frame.
> > > * Allows easy linking, referencing and footnoting; critical for
> > >   Scientific literature.
> > > * Can be rendered into a matching document in most important
> > > formats for academic publishing standard: HTML (->Blog/Journal),
> > > LaTeX (->PDF)
> > > * Has extensive software support already; don't shave yaks!
> > >
> > > ## Markdown Extra
> > > I would like to suggest that Markdown, particularly a flavour of
> > > Markdown Extra, would be ideal. If you're not familiar with the
> > > Markdown syntax, it might hearten you to know that _this email is
> > > written in markdown_. It was specifically designed to mimic how
> > > people place emphasis, build lists, write titles and otherwise
> > > structure text to be readable in the absence of pretty rendering..
> > > except that it allows you to then render that into a variety of
> > > formats.
> > >
> > > Markdown normally renders into HTML, but there are LaTeX renderers
> > > and Wikimedia-flavour wikitext renderers, and pandoc allows you to
> > > render into anything from RTF to .doc, if you like. There are a
> > > bunch of scripts and applications that specialise in posting
> > > markdown documents to Wordpress blogs (which account for a
> > > two-figure percentage of all known websites at present, so that's
> > > not to be sniffed at).
> > >
> > > [Markdown Extra](http://michelf.ca/projects/php-markdown/extra/)
> > > features referencing (in the form of footnotes), abbreviations,
> > > and tables, all using fairly common-sense syntax. Markdown Extra
> > > is supported in PHP and in Python's Markdown package (if the
> > > "extra" plugin is loaded).
> > >
> > > ## Infrastructure
> > > To store and manage publications, given the flexibility, you could
> > > have a wiki or blog to publish rendered documents, and a more
> > > formal repository that stores the Markdown source files and
> > > associated images. That would cater to both ends of the technical
> > > spectrum; those that just want a clear way to present data, and
> > > those that want crunchy machine-parsable everything for meta
> > > stuff.
> > >
> > > For DIYbio purposes, it would be pretty straightforward to knock
> > > together a micro-publishing system where people submit their
> > > papers, they are put up on a github[^1] repo where people can
> > > offer issues and feedback (you don't need to know git to use the
> > > issues feature, so it's non-techie safe) on the content of the
> > > document. When it passes muster/review, it can be script-rendered
> > > and published to wordpress and/or a wiki automatically.
> > >
> > > Given that many people will already have dabbled with Markdown,
> > > and those that haven't can already read it legibly (this email as
> > > case-in-point), and the ease with which it could be rolled out and
> > > diversified, I'd be strongly in favour of choosing it over
> > > something over-lightweight and over-centralised like just-wiki,
> > > or overly technical and offputting like LaTeX-and-git-only
> > >
> > > [^1]: I say github and not gitorious because github has better
> > > support for Markdown, sadly, and their issues interface is
> > > friendlier.
> > >
> > > ## Citizen Science Quarterly
> > > With regard to the other publishing-relevant thread from Jacob,
> > > the CSQ might be rebooting soon (!), and it would probably make
> > > Jacob's job a lot easier if submissions could be trivially
> > > rendered into high-quality HTML and LaTeX, for the online and
> > > print(?) editions respectively.
> > >
> > > Thoughts?
> > >
> > > On Tue, 13 Aug 2013 13:11:54 -0700
> > > Nathan McCorkle <nmz787@gmail.com> wrote:
> > >
> > > > Lots of journals already offer LaTex templates... so what's the
> > > > best interface to magically transform plaintext blocks? A web
> > > > form with blank text boxes and a submit button works, but it
> > > > doesn't let people come back and edit it. Bryan says mediawiki
> > > > (wikipedia) is hard to parse by a computer, but being a wiki
> > > > which /does/ some formatting, the wiki engine /must/ be parsing
> > > > it. I don't think mediawiki formatting is necessary though if we
> > > > had a button to shove it all into a PDF... the wiki would just
> > > > serve as a way to enter and edit the text. Leave the wiki
> > > > unformatted save for some keywords to delineate different blocks
> > > > that are needed for the LaTex template.
> > > >
> > > > On Tue, Aug 13, 2013 at 12:45 PM, Sebastian Cocioba
> > > > <scocioba@gmail.com> wrote:
> > > > > If someone with a knack for aesthetics can make a template and
> > > > > we can vote on it? Try to get the ball rolling?
> > > > >
> > > > > Sent from my Windows Phone From: Nathan McCorkle
> > > > > Sent: 8/13/2013 1:44 PM
> > > > > To: diybio
> > > > > Subject: Re: [DIYbio] Re: Standardized DIYbio report format?
> > > > > WAS: Endophyte isolation and first successful sequencing
> > > > > Maybe a template wiki page would suffice? You copy the
> > > > > template to your own wiki page, then swap the template
> > > > > sections for your words. Wikis are version controlled so
> > > > > edits are tracked, and there are already collaboration tools
> > > > > built-in to each page (see wikipedia's 'talk' page that goes
> > > > > along with each article).
> > > > >
> > > > > You can convert any text into a PDF format using some code,
> > > > > there'd probably need to be a stylistic format chosen for this
> > > > > template.
> > > > >
> > > > > Maybe it'd be best to pick a stylistic format from one of the
> > > > > current science publishers (they're all OK, though I don't
> > > > > really care for 2 columns of text), then work out the wiki
> > > > > template and the PDF generation stuff.
> > > > >
> > > > > --
> > > > > -- You received this message because you are subscribed to the
> > > > > Google Groups DIYbio group. To post to this group, send email
> > > > > to diybio@googlegroups.com. To unsubscribe from this group,
> > > > > send email to diybio+unsubscribe@googlegroups.com. For more
> > > > > options, visit this group at
> > > > > https://groups.google.com/d/forum/diybio?hl=en Learn more at
> > > > > www.diybio.org ---
> > > > > You received this message because you are subscribed to the
> > > > > Google Groups "DIYbio" group.
> > > > > To unsubscribe from this group and stop receiving emails from
> > > > > it, send an email to diybio+unsubscribe@googlegroups.com.
> > > > > To post to this group, send email to diybio@googlegroups.com.
> > > > > Visit this group at http://groups.google.com/group/diybio.
> > > > > To view this discussion on the web visit
> > > > >
> > > https://groups.google.com/d/msgid/diybio/CA%2B82U9JHRRgiFBzNyTd47TNy05gkzJFQWxs_t9kk0jCJ9VSKDA%40mail.gmail.com?hl=en
> > > .
> > > > > For more options, visit
> > > > > https://groups.google.com/groups/opt_out.
> > > > >
> > > > > --
> > > > > -- You received this message because you are subscribed to the
> > > > > Google Groups DIYbio group. To post to this group, send email
> > > > > to diybio@googlegroups.com. To unsubscribe from this group,
> > > > > send email to diybio+unsubscribe@googlegroups.com. For more
> > > > > options, visit this group at
> > > > > https://groups.google.com/d/forum/diybio?hl=en Learn more at
> > > > > www.diybio.org --- You received this message because you are
> > > > > subscribed to the Google Groups "DIYbio" group. To unsubscribe
> > > > > from this group and stop receiving emails from it, send an
> > > > > email to diybio+unsubscribe@googlegroups.com. To post to this
> > > > > group, send email to diybio@googlegroups.com. Visit this group
> > > > > at http://groups.google.com/group/diybio. To view this
> > > > > discussion on the web visit
> > > > >
> > > https://groups.google.com/d/msgid/diybio/-5610509502223320986%40unknownmsgid
> > > .
> > > > > For more options, visit
> > > > > https://groups.google.com/groups/opt_out.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>