Jump to content

User:Monkbot/task 18: cosmetic cs1 template cleanup

From Wikipedia, the free encyclopedia

Monkbot task 18 is a WP:COSMETICBOT task to cleanup cs1|2 templates. It is authorized by Wikipedia:Bots/Requests for approval/Monkbot 18.

cs1|2 templates are transcluded in more than 4.7 million pages. Task 18 is constrained to the canonical list of cs1|2 templates and their more common redirects. Wrapper templates are not considered.

Task 18 does these things:

  1. deletes empty known and unknown named parameters
  2. deletes empty positional parameters
  3. deletes or repairs parameters inside html comments
  4. deletes non-contributing parameter/value pairs
  5. hyphenates cs1|2 parameters when they are written using the to-be-deprecated all-run-together form
  6. converts known language names assigned to |language= to their MediaWiki codes

This task strives to leave template style as it found it so does not change whitespace in and around template parameters. There is an exception to that whitespace rule: blank lines within vertically formatted cs1|2 templates are removed. Though executed by WP:AWB, this task does not perform AWB general fixes.

Task 18 operates only in mainspace. Any article that has {{bots|deny=monkbot 18}} shall be skipped. Additionally, task 18 maintains a list of articles that it will not edit. Contact the bot's operator to get an article added to the list.

Update 2020-12-15: A modified version of the bot (namespace restriction and empty positional parameter portions disabled) is created for use on templates that wrap the cs1|2 canonical templates. This version will be run manually only in template namespace and is identified as task 18a.

Update 2021-01-10: Because the bot depends on results from mw:Help:CirrusSearch to focus its work, an alternate form of the bot task (18b) has been created that focuses on templates that are not cs1|2 templates but are (in many but not all cases) wrapper templates that use a cs1|2 template. Task 18b runs separately from the main task 18.

delete empty parameters

[edit]
cs1|2 templates with empty parameters (estimate)
parameterinitial countcountsearch
{{cite arXiv}}760search
{{cite AV media}}3,1491,008search
{{cite AV media notes}}1,8459search
{{cite bioRxiv}}61search
{{cite book}}88,787 (timed out)7,296 (timed out)search
{{cite citeseerx}}00search
{{cite conference}}1,24611search
{{cite encyclopedia}}8,3934,886search
{{cite episode}}2,882297search
{{cite interview}}636191search
{{cite journal}}111,144 (timed out)1,177 (timed out)search
{{cite magazine}}9,576113search
{{cite mailing list}}300search
{{cite map}}2,6201,835search
{{cite news}}69,918 (timed out)569 (timed out)search
{{cite newsgroup}}410search
{{cite podcast}}28715search
{{cite press release}}2,24637search
{{cite report}}2,32579search
{{cite serial}}540search
{{cite sign}}1113search
{{cite speech}}1012search
{{cite ssrn}}00search
{{cite techreport}}570search
{{cite thesis}}2,35493search
{{cite web}}148,445 (timed out)72,171 (timed out)search
{{citation}}21,644545 (timed out)search
totals477,97390,338 
2020-11-132021-02-03

It is to be expected that some significant portion of cs1|2 transclusions will hold some number of superfluous parameters. One purpose of this task to remove those superfluous parameters. Empty parameters in cs1|2 templates occur for various reasons: the parameter is no longer required or not required in 'this' citation; a skeleton template was copied from a template documentation page and only partially filled in; deprecated error messages were 'fixed' by removing the parameter's value; the template was reworked; the template was placed by automated or semiautomated tools that inserted empty parameters; there are, no doubt, other explanations. For cs1|2, empty parameters serve no purpose.

Editors commonly complain that inline references make it more difficult to read wikitext. Empty parameters in cs1|2 templates occupy space for no meaningful purpose and, as a consequence, contribute to the wikitext readability problem.

The table lists all of the cs1|2 templates. The second column states the approximate number of mainspace articles using the template where at least one parameter was empty. The counts were last updated on the date listed at the bottom of the table.

delete empty positional parameters

[edit]

cs1|2 does not support positional (unnamed) parameters. A positional parameter is one that does not contain an assignment operator (=). Positional parameters that contain nothing or only whitespace, have two forms:

||
|}}

Task 18 deletes the first pipe and any whitespace that exists between it and the succeeding pipe or brace.

Task 18 ignores positional parameters that have content other than whitespace because cs1|2 emits a reader-facing text ignored error message for those sorts of positional parameters.

delete empty named parameters in html comments

[edit]

To cs1|2, parameters that are hidden with html comment markup (<!--content-->) can appear to be empty positional parameters because MediaWiki removes html comments and their content before handing the template to Module:Citation/CS1. cs1|2 templates entirely within html markup are ignored.

Empty parameters that have this form are deleted along with the html comment markup:

|<!-- param= -->

Similarly, empty parameters inside html comment markup are deleted; parameters with assigned values are retained:

<!-- |param=value |param= --> → <!-- |param=value -->

repair parameters in html comments

[edit]

To cs1|2, parameters that are hidden with html comment markup (<!--content-->) can appear to be empty positional parameters because MediaWiki removes html comments and their content before handing the template to Module:Citation/CS1. cs1|2 templates entirely within html markup are ignored.

Parameters that have this form are repaired by moving the preceding pipe into the html comment markup:

|<!-- param=value --> → <!-- |param=value -->

A similar configuration is commonly found. This form may or may not look like an empty positional parameter to cs1|2 depending on the surrounding wikimarkup. This form is repaired for the purposes of consistency:

|<!-- param=value |--> → <!-- |param=value --> |

Comments that do not have an assignment operator (=) but which occupy all of the space between a pair of pipes or a pipe and the template's closing brace are repaired by moving the leading pipe inside the html comment:

|<!-- plain text html comment --> | → <!-- |plain text html comment --> |

delete non-contributing parameters

[edit]

There are several parameters and specific parameter/value pairs that, under certain conditions, to not contribute the rendering of the template. Task 18 deletes these:

  • |deadurl= – is routinely added (with the assigned value y) to cs1|2 templates created by the unmaintained tool ReFill. Because |deadurl= is no-longer supported by cs1|2, when it is present in cs1|2 templates, cs1|2 emits the error message: Unknown parameter |deadurl= ignored (|url-status= suggested) and adds the article to Category:Pages with citations using unsupported parameters which gnomes keep mostly empty. When task 18 encounters |deadurl=y (or the similar |deadurl=yes and |deadurl=true and their hyphenated forms), the parameter shall be deleted. Replacement is not necessary because |deadurl=y (when it was supported) was redundant to the cs1|2 default state. See |url-status= documentation.
  • |mode= – this parameter accepts one of two keywords: cs1 or cs2. The purpose is to direct the template to render in the specified style. For example, {{citation}} is a native cs2 template. {{citation}} will render as if it were a cs1 template when given |mode=cs1; similarly, cs1 templates render in cs2 style when given |mode=cs2. Setting |mode= to the template's native style does not change the rendering. Task 18 deletes |mode= parameters when they match the template's native style. See |mode= documentation.
  • |name-list-style= – controls how name lists are rendered. When assigned one of the values amp, ampersand, and, &, or serial, cs1|2 inserts an ampersand or the word 'and' between the last two names of a name list. When none of the name lists have two or more names, the parameter does not change the rendering. When assigned the value vanc, name lists are rendered in Vancouver style. |name-list-style=vanc requires |last= and |first= (or aliases); without |first= aliases, the parameter does not change the rendering. See |name-list-style= documentation.
  • |<name-list>-link=, |<name-list>-mask=[a] – these wikilink or mask a <name> in an associated <name-list>. When there is no associated |<name>= (for example, |author-mask14=2 but no |author14=), these parameters do not change the rendering.
  • |no-pp= – suppresses p. and pp. page annotation when either of |page= or |pages= has a value; when there is no |page= or |pages= parameter, this parameter does not change the rendering. When found in {{citation}} templates where |journal= has an assigned value or when found in {{cite journal}}, |no-pp= has no value because pagination in journal citations does not use the P. and pp. annotation.
  • |orig-year= – a free-form parameter that accompanies |year=, |date= or when neither of those are present, |publication-date=. When none of those are present, |orig-year= is not rendered.
  • |postscript= – specifies the rendered template's terminal punctuation. Default terminal punctuation for cs1 is a dot; for cs2, terminal punctuation is omitted. |postscript=. in cs1 templates and |postscript=none in cs2 templates does not change the rendering. The form |postscript=<!--none--> is ignored by cs1|2 templates because the html comments make |postscript= an empty parameter.
  • |ref=harv – At the 2020-04-18 update to Module:Citation/CS1 (this edit), |ref=harv was internally defined as the default state so the explicit |ref=harv does not change the template's function. Use of this parameter/value pair is tracked in Category:CS1 maint: ref=harv‎. See |ref= documentation.
  • |url-status= – like its predecessors |dead-url= and |deadurl=, |url-status= is a display control parameter that determines how a cs1|2 template will be rendered when that template has |archive-url= and |archive-date=. When |archive-url= and |archive-date= are empty or omitted, |url-status= is non-functional. See |url-status= documentation.

hyphenate cs1|2 parameter names

[edit]
cs1|2 parameters to be hyphenated
parameterinitial countcountsearch
|accessdate=2,804,8811,841,029search
|archivedate=837,655444,873search
|archiveurl=834,802444,601search
|authorlink=268,23285,737search
|authormask=40search
|booktitle=3,34528search
|chapterurl=17,289250search
|conferenceurl=†4270search
|contributionurl=†400search
|displayauthors=11search
|editorlink=10search
|editormask=10search
|episodelink=2,5392search
|laydate=†1,04913search
|laysource=†1,09113search
|layurl=†5382search
|mailinglist=37976search
|mapurl=8131search
|nopp=2,90235search
|notracking=11search
|origyear=44,0229,841search
|publicationdate=54275search
|publicationplace=1890search
|sectionurl=†654search
|serieslink=5,0428search
|seriesno=†1,07513search
|subjectlink=00search
|timecaption=†390search
|titlelink=†3,51893search
|transcripturl=9101search
|author[0-9]+link=648 (timed out)126 (timed out)search
|author[0-9]+mask=0 (timed out)0 (timed out)search
|editor[0-9]+link=20 (timed out)0 (timed out)search
|editor[0-9]+mask=0 (timed out)0 (timed out)search
|subject[0-9]+link=0 (timed out)0 (timed out)search
totals4,831,3282,826,853 
2020-11-132021-02-03
† currently deprecated
support withdrawn for parameters that are struckout

Because cs1|2 is an amalgam of several individually developed templates, it acquired a variety of parameter-name styles: |lowercase=, |Capitalized=, |camelCase=, |underscore_separated=, |space separated=, |hyphen-separated=, |allruntogether=. The |Capitalized=, |camelCase=, |underscore_separated=, and |space separated= parameter name styles have all been deprecated and support for these styles withdrawn in favor of the |lowercase= and |hyphen-separated= forms as a result of this RfC. For parameter names that are multiword, cs1|2 is gradually shifting to prefer the hyphenated form. The table lists the all-run-together form with the approximate number of articles that transclude cs1|2 templates using these parameters. Task 18 replaces the all-run-together forms of these parameters with the hyphenated forms.

The |accessdate= and |authorlink= parameter names have been included in awb's genfixes since this edit. There has been some pushback against using genfixes to normalize cs1|2 parameter name style. See Wikipedia talk:AutoWikiBrowser/Archive 32 § Citation parameter renaming.

There are commonly used tools that continue to insert the all-run-together names. An interface protected edit request has been submitted to normalize all-run-together parameter names emitted by WP:RefToolbar. See MediaWiki talk:RefToolbarConfig.js § normalize parameter names.

convert language names to codes

[edit]

As a courtesy to editors at other-language wikis, it is desirable to use the MediaWiki language codes instead of English-language language names in |language= so that cs1|2 can render language names using the other wiki's language without the need for an editor to make a translation. Task 18 will replace language names with the associated MediaWiki code. See the |language= documentation and the list of (mostly) English-language language names and associated codes.

edit summary messaging

[edit]

For each edit made, task 18 creates an edit summary detailing what was done. For example, this edit summary from a test (no save) of Ida B. Wells:

Task 18 (cosmetic) (dev test): eval 146 templates: del empty params (1251×); hyphenate params (4×); del pos params (1×); del |ref=harv (4×);

Each summary always begins with the 'Task 18 (cosmetic) (dev test): eval n templates:' leader where n is the total number of cs1|2 templates and recognized redirects that task 18 evaluated. The remaining portion of the edit summary is assembled from one or more of these:

del cmtd params (n×); – deleted commented parameters
rep cmtd params (n×); – repaired commented parameters
del empty params (n×); – deleted empty parameters
hyphenate params (n×); – hyphenated parameter names
del empty pos params (n×); – deleted empty positional parameters
del |ref=harv (n×); – deleted |ref=harv
del |mode= (n×); – deleted |mode=
del |postscript= (n×); – deleted |postscript=
del |url-status= (n×); – deleted |url-status=
del |deadurl= (n×); – deleted |deadurl=
del |no-pp= (n×); – deleted |no-pp=
del |orig-year= (n×); – deleted |orig-year
del |name-list-style= (n×); – deleted |name-list-style=
del |<name-list>-mask= (n×); – deleted |<name-list>-mask=
del |<name-list>-link= (n×); – deleted |<name-list>-link=
cvt lang vals (n×); – converted language values
skip (n×); – during development of task 18, it found malformed cs1|2 templates that contain the opening {| or closing |} that belongs to wikitables. An example of this is found in this version of Satish Shah which is missing the closing }} for this reference. Left unchecked, task 18 will see the table content as part of that broken {{cite news}} template and delete many of the double pipe markup used to separate cells in the wikitable. This particular garbage is detected and skipped but editors are endlessly clever when it comes to breaking cs1|2 templates.

AWB truncates long edit summaries (phab:T199347) so the messaging is necessarily terse in an attempt to list everything accomplished by task 18 in the summary. When that is not possible, task 18 truncates its summary and adds an ellipsis.

known issues

[edit]

When an article transcludes a named reference that exactly matches a named reference in the article, and task 18 edits the article's named reference, MediaWiki will emit this error message:

The named reference "$1" was defined multiple times with different content (help page).

Task 18 cannot know that a cs1|2 template that it edits is mirrored in a transcluded template. If you know of articles where this is a problem, give a list of those articles to the bot's operator so that the bot can be instructed to skip those articles.

notes

[edit]
a. ^ The |<name-list>-link= and |<name-list>-mask= parameters are:
  • |author-link=
  • |authorn-link=
  • |author-linkn=
  • |contributor-link=
  • |contributorn-link=
  • |contributor-linkn=
  • |editor-link=
  • |editorn-link=
  • |editor-linkn=
  • |interviewer-link=
  • |interviewern-link=
  • |interviewer-linkn=
  • |translator-link=
  • |translatorn-link=
  • |translator-linkn=
  • |author-mask=
  • |authorn-mask=
  • |author-maskn=
  • |contributor-mask=
  • |contributorn-mask=
  • |contributor-maskn=
  • |editor-mask=
  • |editorn-mask=
  • |editor-maskn=
  • |interviewer-mask=
  • |interviewern-mask=
  • |interviewer-maskn=
  • |translator-mask=
  • |translatorn-mask=
  • |translator-maskn=

script

[edit]
// This script does cosmetic work:
//		repairs / removes commented parameters
//		deletes empty parameters
//		replaces un-hyphenated parameters with their hyphenated forms
//		deletes |ref=harv
//		replaces language names in |language= with their MediaWiki ISO 639-like codes
//
// wikitext searches:
//	html comments:
//		hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]it[ae][^\}]*\| *\]+\-\-+\>/
//	mode:
//		hastemplate:Citation insource:/\{ *[Cc]itation[^\}]*\| *mode *= *cs2/
//		hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]ite[^\}]*\| *mode *= *cs1/
//	deadurl:
//		hastemplate:"Module:Citation/CS1" insource:"deadurl"
//	postscript:
//		hastemplate:Citation insource:/\{ *[Cc]itation[^\}]*\| *postscript *= *none/
//		hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]ite[^\}]*\| *postscript *= *\./
//		hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]it[ae][^\}]*\| *postscript *= *\
//
// category:
//		CS1 errors: empty unknown parameters‎
//


public string ProcessArticle(string ArticleText, string ArticleTitle, int wikiNamespace, out string Summary, out bool Skip)
	{
 	if (0 != wikiNamespace )										// if  is not mainspace
		{
		if (-1 == ArticleTitle.IndexOf ("User:Trappist the monk"))	// don't skip pages in my own user space
			{
			Skip = true;
			Summary = @"article not in mainspace: " + wikiNamespace;
			return ArticleText;										// abandon this edit
			}
		}

	if (skip_map.ContainsKey (ArticleTitle))						// if  is in the skip list
		{
		Skip = true;
		Summary = @"article in skip list";
		return ArticleText;											// abandon this edit
		}

	Skip = false;													// assume that something will be changed

//	Summary = "[[User:Monkbot/task 18|Task 18 (cosmetic)]] (dev test):";	// uses redirect to User:Monkbot/task 18: cosmetic cs1 template cleanup
//	Summary = "[[User:Monkbot/task 18|Task 18 (cosmetic)]] (BRFA trial):";
	Summary = "[[User:Monkbot/task 18|Task 18 (cosmetic)]]:";

	int		total_count = 0;										// total number of cs1|2 templates
	int		skip_count = 0;											// skip count incremented when cite template holds wikitable markup
	int		comment_deleted = 0;
	int		comment_repaired = 0;
	int		empty_positional_deleted = 0;
	int		params_removed_count = 0;
	int		params_hyphenated_count = 0;
	int		ref_harv_count = 0;
	int		url_status_count =0;
	int		deadurl_count = 0;
	int		nopp_count = 0;
	int		orig_count = 0;
	int		mode_count = 0;
	int		ps_count = 0;
	int		nls_count = 0;
	int		mask_count = 0;
	int		link_count = 0;
	int		language_count = 0;


//---------------------------< B E G I N >--------------------------------------------------------------------

	ArticleText = hide (ArticleText, IS_CS1);						// hide all templates except cs1|2 and hide wikilinks


//---------------------------< M A I N >----------------------------------------------------------------------
//
// this is main
//

	if (Regex.Match (ArticleText, template_pattern).Success)
		ArticleText = Regex.Replace (ArticleText, template_pattern,
			delegate(Match match)
				{
				string	template = match.Groups[0].Value;					// this will be returned if no changes

				if ((-1 != template.IndexOf("__0T4BL3__")) || (-1 != template.IndexOf("__CT4BL3__")))	// if table markup found in the cite template abandon
					{
					skip_count++;
					return template;
					}

				total_count++;												// bump total number of cs1|2 templates tally

				template = counted_replace (template, comment_params_1_pattern, "$2$1$3$5$4", ref comment_repaired);		// | to |
				template = counted_replace (template, comment_params_2_pattern, "$2$1$3", ref comment_repaired);			// | to 
				template = counted_replace (template, comment_params_3_pattern, "", ref params_removed_count);				// | to (delete)
				template = counted_replace (template, comment_params_4_pattern, "$1", ref params_removed_count);			//  to 
				template = counted_replace (template, comment_params_5_pattern, "$2$1$3", ref comment_repaired);			// | to 

				template = counted_replace (template, empty_positional_parameter_nl, "$1", ref empty_positional_deleted);	// || or |} to | or } (newline variant)
				template = counted_replace (template, empty_positional_parameter, "$1", ref empty_positional_deleted);		// || or |} to | or }

				template = counted_replace (template, ref_harv_pattern, "", ref ref_harv_count);							// |ref=harv to (delete)

				template = mode_remove (template, ref mode_count);

				template = postscript_remove (template, ref ps_count);

				template = empty_param_remove (template, ref params_removed_count);							// this is a prerequisite function for these:
					template = url_status_remove (template, ref url_status_count, ref deadurl_count);
					template = hyphenate (hyphenate_map, template, ref params_hyphenated_count);			// this is a prerequisite function for these:
						template = nopp_remove (template, ref nopp_count);
						template = orig_year_remove (template, ref orig_count);
						template = name_list_style_remove (template, ref nls_count);
						template = name_link_mask_remove (template, ref mask_count, @"mask");
						template = name_link_mask_remove (template, ref link_count, @"link");

				template = language (language_map, template, ref language_count);

				while (Regex.Match (template, empty_comment_pattern).Success)								// cleanup; delete  (empty or only whitespace)
					template = Regex.Replace (template, empty_comment_pattern, "");

				return template;
				});


//---------------------------< F I N I S H >------------------------------------------------------------------

	ArticleText = unhide (ArticleText);										// unhide all that is hidden

	if (0 != total_count)													// build our edit summary
		{
		Summary = summary_concat (Summary, " eval " + total_count + " template" + (1 == total_count ? ":" : "s:"));

		if (0 != comment_deleted)
			Summary = summary_concat (Summary, " del cmtd params (" + comment_deleted + "×);");

		if (0 != comment_repaired)
			Summary = summary_concat (Summary, " rep cmtd params (" + comment_repaired + "×);");

		if (0 != params_removed_count)
			Summary = summary_concat (Summary, " del empty params (" + params_removed_count + "×);");

		if (0 != params_hyphenated_count)
			Summary = summary_concat (Summary, " hyphenate params (" + params_hyphenated_count + "×);");

		if (0 != empty_positional_deleted)
			Summary = summary_concat (Summary, " del pos params (" + empty_positional_deleted + "×);");

		if (0 != ref_harv_count)
			Summary = summary_concat (Summary, " del |ref=harv (" + ref_harv_count + "×);");

		if (0 != mode_count)
			Summary = summary_concat (Summary, " del |mode= (" + mode_count + "×);");

		if (0 != ps_count)
			Summary = summary_concat (Summary, " del |postscript= (" + ps_count + "×);");

		if (0 != url_status_count)
			Summary = summary_concat (Summary, " del |url-status= (" + url_status_count + "×);");

		if (0 != deadurl_count)
			Summary = summary_concat (Summary, " del |deadurl= (" + deadurl_count + "×);");

		if (0 != nopp_count)
			Summary = summary_concat (Summary, " del |no-pp= (" + nopp_count + "×);");

		if (0 != orig_count)
			Summary = summary_concat (Summary, " del |orig-year= (" + orig_count + "×);");

		if (0 != nls_count)
			Summary = summary_concat (Summary, " del |name-list-style= (" + nls_count + "×);");

		if (0 != mask_count)
			Summary = summary_concat (Summary, " del |-mask= (" + mask_count + "×);");

		if (0 != link_count)
			Summary = summary_concat (Summary, " del |-link= (" + link_count + "×);");

		if (0 != language_count)
			Summary = summary_concat (Summary, " cvt lang vals (" + language_count + "×);");

		if (0 != skip_count)
			Summary = summary_concat (Summary, " skip (" + skip_count + "×);");
		}

	else
		{
		Skip = true;
		Summary = Summary + " no templates to evaluate;";
		}

	System.IO.StreamWriter sw;
	string	log_file = @"Z:\Wikipedia\AWB\Monkbot_tasks\Task18_" + DateTimeOffset.Now.ToString("u").Substring (0, 10) + @".csv";

	if (!System.IO.File.Exists (log_file))
		{														// for csv import, pipe is string delimiter here because not legal wp title char
		sw = System.IO.File.AppendText (log_file);
		sw.WriteLine ("|Article|, |evaluated|, |deleted comment|, |repaired comment|, |empty params|, |hyphenated|, |empty pos params|, |ref=harv|, |mode|, |postscript|, "
		+ "|url-status|, |deadurl|, |no-pp|, |orig-year|, |name-list-style|, |-mask|, |-link|, |language|, |skipped|");
		sw.Close();
		}

	if (Skip)
		{
		sw = System.IO.File.AppendText (log_file);
		sw.WriteLine ("|" + ArticleTitle + "|," + total_count);
		sw.Close();
		}
	else
		{
		sw = System.IO.File.AppendText (log_file);
		sw.WriteLine ("|" + ArticleTitle + "|," + total_count + "," + comment_deleted + "," + comment_repaired + "," + params_removed_count + "," + params_hyphenated_count
		+ "," + empty_positional_deleted + "," + ref_harv_count + "," + mode_count + "," + ps_count + "," + url_status_count + "," + deadurl_count
		+ "," + nopp_count + "," + orig_count + "," + nls_count + "," + mask_count + "," + link_count + "," + language_count + "," + skip_count);
		sw.Close();
		}

	return ArticleText;
	}


//===========================<< S U P P O R T >>==============================================================

//---------------------------< H I D E >----------------------------------------------------------------------
//
// HIDE TEMPLATES: find templates that are not ; replace the opening {{ with __0P3N__, the closing }}
// with __CL0S3__, and internal | (pipes) with __P1P3__
//
// single curly braces in urls and other parameter values can confuse other regex in this code so replace {
// with __0CU!21Y__ and } with __CCU!21Y__
//

private string hide (string ArticleText, string dont_hide)
	{
	string pattern = @"\{\{(?!\s*" + dont_hide + @")[^\{\}]*\}\}";
	if (Regex.Match (ArticleText, pattern).Success)
		{
		ArticleText = Regex.Replace(ArticleText, pattern,
			delegate(Match match)
				{
				string	fixed_template;										// a hidden template is assembled here
				string	raw_template = match.Groups[0].Value;				// the whole template

				pattern = @"\{\{";											// hide the opening {{
				fixed_template = Regex.Replace (raw_template, pattern, "__0P3N__");

				pattern = @"\}\}";											// hide the closing }}
				fixed_template = Regex.Replace (fixed_template, pattern, "__CL0S3__");

				pattern = @"\|";											// and hide the pipes
				fixed_template = Regex.Replace (fixed_template, pattern, "__P1P3__");

				return fixed_template;
				});
		}

	pattern = @"(\\|\}]*)\{\{(\s*" + dont_hide + @"[^\}]*)\}\}([^\>]*\-{2,}\>)";		// 
	ArticleText = Regex.Replace(ArticleText, pattern, "$1__0P3N__$2__CL0S3__$3");

	pattern = @"\{\|";														// open table markup
	ArticleText = Regex.Replace(ArticleText, pattern, "__0T4BL3__");

	pattern = @"\|\}(?!\})";												// open table markup
	ArticleText = Regex.Replace(ArticleText, pattern, "__CT4BL3__");

	pattern = @"([^\{])\{([^\{])";											// single opening curly brace
	ArticleText = Regex.Replace(ArticleText, pattern, "$1__0CU!21Y__$2");

	pattern = @"([^\}])\}([^\}])";											// single closing curly brace
	ArticleText = Regex.Replace(ArticleText, pattern, "$1__CCU!21Y__$2");

	pattern = @"\[\[(?![Ff]ile|[Ii]mage)([^\|\]]+)\|([^\]]+)\]\]";			// HIDE complex wikilinks: [[article title|label]] to __WL1NK_O__article title__P1P3__label__WL1NK_C__
	ArticleText = Regex.Replace(ArticleText, pattern, "__WL1NK_O__$1__P1P3__$2__WL1NK_C__");	// [[File: with wikilinks inside can be confusing

	pattern = @"\[\[([^\]]+)\]\]";											// HIDE simple wikilinks: [[article title]] to __WL1NK_O__article title__WL1NK_C__
	ArticleText = Regex.Replace(ArticleText, pattern, "__WL1NK_O__$1__WL1NK_C__");

	return ArticleText;
	}


//---------------------------< U N H I D E >------------------------------------------------------------------
//
// UNHIDE TEMPLATES: find templates and wikilinks that are hidden; replace the 'hide' keywords with the
// appropriate wiki markup
//

private string unhide (string ArticleText)
	{
	ArticleText = Regex.Replace(ArticleText, @"__WL1NK_O__", "[[");		// UNHIDE: replace __WL1NK_O__ with [[
	ArticleText = Regex.Replace(ArticleText, @"__WL1NK_C__", "]]");		// UNHIDE: replace __WL1NK_C__ with ]]
	ArticleText = Regex.Replace(ArticleText, @"__P1P3__", "|");			// UNHIDE: replace __P1P3__ with |

	ArticleText = Regex.Replace(ArticleText, @"__0T4BL3__", "{|");		// UNHIDE: replace __0T4BL3__ with {|
	ArticleText = Regex.Replace(ArticleText, @"__CT4BL3__", "|}");		// UNHIDE: replace __CT4BL3__ with |}

	ArticleText = Regex.Replace(ArticleText, @"__0CU!21Y__", "{");		// UNHIDE: replace __0CU!21Y__ with {
	ArticleText = Regex.Replace(ArticleText, @"__CCU!21Y__", "}");		// UNHIDE: replace __CCU!21Y__ with }

	ArticleText = Regex.Replace(ArticleText, @"__0P3N__", "{{");		// UNHIDE: replace __0P3N__ with {{
	ArticleText = Regex.Replace(ArticleText, @"__CL0S3__", "}}");		// UNHIDE: replace __CL0S3__ with }}

	return ArticleText;
	}


//---------------------------< S U M M A R Y _ C O N C A T >--------------------------------------------------
//
// concatenates text onto an existing edit summary string, limiting the string to a length of no more than 247
// characters.  When  appended with  would be longer than the allowed 247 character limit, this
// function replaces  with an ellipsis.  Once an ellipsis is added, no more  can be added to 
//

private string summary_concat (string summary, string text)
	{
	if (0 <= summary.IndexOf ("..."))									// if ellipsis already present in , abandon
		return summary;

	if (247 >= (summary.Length + text.Length + 3))						// if adding  to summary will overrun the 247 char limit (+ 3 to make sure we can add ellipsis if necessary)
		return summary + text;											// append  to  and done

	return summary + "...";												// append ellipsis instead
	}
	

//---------------------------< C O U N T E D _ R E P L A C E >------------------------------------------------
//
// common function to replace  with  and bump  until no more 
//

private string counted_replace (string template, string pattern, string replace, ref int count)
	{
	Regex rgx = new Regex (pattern);											// make a new regex from 

	while (Regex.Match (template, pattern).Success)								// look for  in